Systems and methods for analyzing and displaying acoustic data

ABSTRACT

Some systems include an acoustic sensor array configured to receive acoustic signals, an electromagnetic imaging tool configured to receive electromagnetic radiation, a user interface, a display, and a processor. The processor can receive electromagnetic data from the electromagnetic imaging tool and acoustic data from the acoustic sensor array. The processor can generate acoustic image data of the scene based on the received acoustic data, generate a display image comprising combined acoustic image data and electromagnetic image data, and present the display image on the display. The processor can receive an annotation input from the user interface and update the display image based on the received annotation input. The processor can be configured to determine one or more acoustic parameters associated with the received acoustic signal and determine a criticality associated with the acoustic signal. A user can annotated the display image with determined criticality information or other determined information.

RELATED MATTERS

This application claims priority to U.S. Patent Application No.62/702,716, filed Jul. 24, 2018, the entire contents of which areincorporated herein by reference.

BACKGROUND

Presently available acoustic imaging devices include acoustic sensorarray configurations that have various frequency sensitivity limitationsdue to a variety of factors. For instance, some acoustic imaging devicesare designed to be responsive to a range of acoustic frequencies betweenapproximately 20 Hz and approximately 20 kHz. Other devices (e.g.,ultrasonic devices) are designed to be responsive to a range of acousticfrequencies between approximately 38 kHz and approximately 45 kHz.

However, acoustic imaging devices that are generally designed operatingin the 20 Hz to 20 kHz frequency range cannot effectively detect orimage higher frequencies, for example, up to or above approximately 50kHz. Likewise, acoustic or ultrasonic devices that are designed tooperate in the 20 kHz to 50 kHz frequency range cannot effectivelydetect and/or image lower frequencies, for example, at or below 20 kHz.This can be for a variety of reasons. For example, sensor arrays whichare optimized for lower (e.g., audible) frequencies typically containindividual sensors that are farther apart than do sensor arrays that areoptimized for higher (e.g., ultrasonic) frequencies.

Additionally or alternatively to hardware concerns, differentcalculation algorithms and methods of acoustic imaging are often bettersuited for acoustic signals having different frequencies and/ordifferent distances to target, making it difficult to determine how tobest to acoustically image a scene without, particularly to aninexperienced user.

Such discrepancies in imaging different acoustic frequency ranges aredue, in part, to the physics behind the propagation of sound waves ofdifferent frequencies and wavelengths through air. Certain arrayorientations, array sizes, and calculation methods can generally bebetter suited for acoustic signals having different frequencycharacteristics (e.g., audible frequencies, ultrasonic frequencies,etc.).

Similarly, different array properties and/or calculation methods can bebetter suited for acoustic scenes at different distances to target. Forexample, near field acoustic holography for targets at very closedistances, various acoustic beamforming methods for targets at greaterdistances.

Accordingly, acoustic inspection using acoustic arrays (e.g., foracoustic imaging) can require a wide range of equipment, for example,for analysis of acoustic signals having different frequency ranges aswell as expertise in understanding when different hardware andcalculation techniques are appropriate for performing acoustic analysis.This can make acoustic inspections time- and cost-intensive, and canrequire an expert to perform such inspections.

For example, a user may be forced to manually select various hardwareand/or software for performing acoustic analysis. However, aninexperienced analyst may be incapable of knowing the preferredcombination of hardware and software for a given acoustic analysisand/or acoustic scene. Additionally, isolating a sound of interest fromwithin a scene can provide its own challenges, particularly in acluttered scene, and may prove tedious and frustrating to aninexperienced user. For instance, a given acoustic scene, especially ina noisy environment, can include acoustic signals including any numberof frequency, intensity, or other characteristics that may obscureacoustic signals of interest.

Traditional systems often require users to manually identify variousacoustic parameters of interest prior to inspection in order to analyzethe sounds of interest. However, an inexperienced user may be unaware ofhow to best isolate and/or identify various sounds of interest.

Additionally, when multiple imaging technologies (e.g., visible light,infrared, ultraviolet, acoustic, or other imaging techniques) are usedin tandem while inspecting the same object or scene, the physicalplacement and or other settings (e.g., focus position) of the tools usedto perform the different imaging techniques can impact the analysis. Forexample, different locations and/or focus positions of each imagingdevice can result in a parallax error wherein the resulting images maybe misaligned. This may result in inability to properly localize areasof interest and/or problem areas within a scene, documentation errors,and misdiagnosis of problems. For example, with respect to acousticimage data, it can be difficult to identify a location or source of anacoustic signal of interest if acoustic image data is misaligned withrespect to image data from other imaging technologies (e.g., visiblelight and/or infrared image data).

Existing ultrasonic test and inspection tools employ ultrasonicsensor(s), with or without the use of a parabolic dish in order toassist in focusing the sound towards the receiving sensor(s). When asound of a specific frequency is detected, it is typically displayed asa rising or falling numerical value, or on a frequency or decibel levelgraph on the display of the device. This can be very confusing andnon-intuitive to many users. No image of the live scene orvisualizations of the sounds are available.

Isolating, localizing, and analyzing a specific sound can be a tediousprocess, and can be confusing for many end users. The complex andnon-intuitive interface between device and human can become a barrier toeffective use of the device, and/or require the need for additionaltraining even to operate basic functionality on the device.

Advanced acoustic imaging devices have the capability of producing afalse-color visual representation of sounds integrated with a still orlive visible image of a scene. Even on these devices, selection andadjustment controls are important for proper visualization of sounds.However, traditional controls have been developed for use byhighly-trained acoustics technicians and specialists. These controls areoften non-intuitive to the average user, and can result in someconfusion over proper selection and visualization parameter controls.Use of these controls by those with a lower level of training can becumbersome, and lead to errors in parameter selection, and ultimatelylead to poor acoustic visualizations.

Moreover, additional contextual information is often needed with thismethod as well, in order to perform proper analysis and reportingactivities. Technicians desiring to collect additional contextualinformation about a scene that is being inspected with a traditionalultrasonic testing device or an acoustic imager typically must takephotographs with a separate camera or device and/or record writtennotes, or notes recorded in a separate device such as a PC, tablet,smartphone, or other mobile device. These secondary notes must then bemanually synchronized or matched up with the data from the ultrasonictool or acoustic imager. This can take a significant amount of time, andcan also be prone to errors in matching the correct data with thecorresponding secondary contextual information.

SUMMARY

Some aspects of the disclosure are directed toward an acoustic analysissystem. Systems can include an acoustic array comprising a plurality ofacoustic sensor elements, each of the plurality of acoustic sensorelements can be configured to receive acoustic signals from an acousticscene and output acoustic data based on the received acoustic signals.

Systems can include an electromagnetic imaging tool configured toreceive electromagnetic radiation from a target scene and outputelectromagnetic image data representative of the receivedelectromagnetic radiation. The electromagnetic imaging tool may beconfigured to detect electromagnetic radiation from a range ofwavelengths, such as a range including the visible light and/ornear-infrared light spectrum. In some systems, the electromagneticimaging system may comprise a visible light camera module and/or aninfrared camera module.

Systems can include a user interface, a display, and a processor. Theprocessor may be in communication with the acoustic sensor array, theelectromagnetic imaging tool, the user interface, and the display.

In some systems, the processor can be configured to receiveelectromagnetic data from the electromagnetic imaging tool and receiveacoustic data from the acoustic sensor array. The processor may alsogenerate acoustic image data of the scene based on the received acousticdata; generate a display image comprising combined acoustic image dataand electromagnetic image data; and present the display image on thedisplay. In some embodiments, the processor may receive an annotationinput from the user interface and update the display image on thedisplay based on the received annotation input. The annotation input maycomprise a freestyle annotation received via a touch screen; a selectionof an icon, or predefined shape, and/or an alphanumeric input.

In some systems, the processor is configured to determine one or moreacoustic parameters associated with the received acoustic signal anddetermine a criticality associated with the acoustic signal, forexample, based on a comparison of the one or more acoustic parameters toone or more predetermined thresholds. In some embodiments, the processormay also update the display image based on the determined criticality. Auser may annotate an image with determined criticality information. Auser may similarly annotate an image with determined information, suchas a distance to target value.

The details of one or more examples set forth in the accompanyingdrawings and the description below. Other features, objects, andadvantages will be apparent from the description and drawings, and fromthe claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B show front and back views of an example acoustic imagingdevice.

FIG. 2 is a functional block diagram illustrating components of anexample of acoustic analysis system.

FIGS. 3A, 3B, and 3C show schematic diagrams of exemplary acousticsensor array configurations within an acoustic analysis system.

FIGS. 4A and 4B show schematic illustrations of parallax error in thegeneration of a frame of visible light image data and acoustic imagedata.

FIGS. 5A and 5B show parallax correction between a visible light imageand an acoustic image.

FIGS. 5C and 5D are colorized versions of FIGS. 5A and 5B.

FIG. 6 is a process flow diagram showing an exemplary method forgenerating a final image combining acoustic image data andelectromagnetic image data.

FIG. 7 is a process-flow diagram showing an exemplary process forgenerating acoustic image data from received acoustic signals.

FIG. 8 shows an exemplary lookup table for determining an appropriatealgorithm and sensor array for use during an acoustic imaging process.

FIG. 9A is an exemplary plot of frequency content of received image dataover time in an acoustic scene.

FIG. 9B shows an exemplary scene including a plurality of locationsemitting acoustic signals.

FIG. 9C shows a plurality of combined acoustic and visible light imagedata at a plurality of predefined frequency ranges.

FIGS. 10A and 10B are exemplary display images including combinedvisible light image data and acoustic image data.

FIGS. 11A and 11B show exemplary plots of frequency vs. time of acousticdata in an acoustic scene.

FIGS. 12A, 12B, and 12C show multiple exemplary ways for comparingacoustic image data to historical acoustic image data stored in adatabase.

FIG. 13 is a process-flow diagram showing exemplary operation ofcomparing received acoustic image data to a database for objectdiagnostics.

FIG. 14 shows a visualization of acoustic data using a gradientpalettization scheme.

FIG. 15 shows a visualization of acoustic data using a plurality ofshaded concentric circles.

FIG. 16 shows an exemplary visualization including both non-numericinformation and alphanumeric information.

FIG. 17 shows another example visualization including both non-numericinformation and alphanumeric information.

FIG. 18 shows another example visualization including both non-numericinformation and alphanumeric information.

FIG. 19 shows an exemplary visualization showing indicators of differentsize and color representative of different acoustic parameter values.

FIG. 20 shows an exemplary visualization showing a plurality ofindicators having different colors indicative of a severity indicated byacoustic signals from the corresponding locations.

FIG. 21 shows a scene including indicators at a plurality of locationswithin the scene showing acoustic signals meeting a predeterminedcondition in a distinguishing way.

FIG. 22 shows a display image including a plurality of icons positionedwithin the display image indicating recognized acoustic profiles withinthe scene.

FIG. 23 shows another exemplary display image showing acoustic data viaa plurality of indicators using concentric circles and alphanumericinformation representing acoustic intensity associated with each of theacoustic signals.

FIG. 24 shows an example display image having an indicator andadditional alphanumeric information associated with the representedacoustic signal.

FIG. 25A shows a system including a display in which an indicator withina display image is selected and a laser pointer emitting a laser towarda scene.

FIG. 25B shows the display shown in the system view of FIG. 25.

FIG. 26 shows a display image including an indicator having a gradientpalettization scheme representing acoustic image data and includingacoustic image blending control.

FIG. 27 shows a display image including an indicator having a concentriccircle palettization scheme representing acoustic image data andincluding acoustic image blending control.

FIG. 28 shows a display image including an indicator having a gradientpalettization indicating a location in the scene meeting one or morefilter conditions.

FIG. 29 shows a display image including an indicator having a concentriccircle palettization indicating a location in the scene meeting one ormore filter conditions.

FIG. 30 shows a display image including two indicators, each having agradient palettization indicating a location in the scene meeting adifferent filter condition.

FIG. 31 shows a display interface including a display image and avirtual keyboard.

FIG. 32 shows a display embedded into eyewear that can be worn by a userand display a display image.

FIGS. 33A and 33B show a dynamic display image including indicatorshaving dynamic intensity based on the pointing of an acoustic sensorarray.

FIG. 34 shows an exemplary display image that provides an indication toa user or technician regarding the potential criticality of and thepotential lost cost due to air leaks identified in the scene.

FIG. 35 shows an example of a user annotating the display image withon-display freeform annotations.

FIG. 36 shows an example of an annotated display image includinginstructions and relevant location information.

FIG. 37 shows an example of a user annotating the display image withon-display icon annotation.

FIG. 38 shows an example of a user annotating the display image with anon-display shape annotation.

FIG. 39 shows an example of an annotated display image includingon-display shape and icon annotations.

FIG. 40 shows an interface including a display image and amulti-parameter data visualization including plurality of frequencyranges on the right-hand side of the display image.

FIG. 41 shows an interface including a display image and amulti-parameter representation of frequency information, includingplurality of frequency ranges positioned along a lower edge of thedisplay image.

FIG. 42 shows a display image including frequency information for aplurality of frequency bands and peak values for a plurality offrequency bands.

FIG. 43 shows a display image including a multi-parameter representationshowing intensity information for a plurality of frequencies.

FIG. 44 shows a multi-parameter representation including palettized setof frequency ranges, wherein the palettization represents a decibelrange into which each frequency range falls.

FIG. 45 shows an example display image including a multi-parameterrepresentation showing different frequency ranges and indicatorspalettized according to severity.

FIG. 46 shows intensity (in dB) vs. time trends for each of a pluralityof frequency ranges in multi-parameter representation on a displayimage.

DETAILED DESCRIPTION

FIGS. 1A and 1B show front and back views of an example acoustic imagingdevice. FIG. 1A shows a front side of an acoustic imaging device 100having a housing 102 supporting an acoustic sensor array 104 and anelectromagnetic imaging tool 106. In some embodiments, the acousticsensor array 104 includes a plurality of acoustic sensor elements, eachof the plurality of acoustic sensor elements being configured to receiveacoustic signals from an acoustic scene and output acoustic data basedon the received acoustic signals. The electromagnetic imaging tool 106can be configured to receive electromagnetic radiation from a targetscene and output electromagnetic image data representative of thereceived electromagnetic radiation. The electromagnetic imaging tool 106can be configured to detect electromagnetic radiation in one or more ofa plurality of ranges of wavelengths, such as visible light, infrared,ultraviolet, or the like.

In the illustrated example, the acoustic imaging device 100 includes anambient light sensor 108 and a location sensor 116, such as a GPS. Thedevice 100 includes a laser pointer 110, which in some embodiments,includes a laser distance meter. The device 100 includes a torch 112,which can be configured to emit visible light radiation toward a scene,and an infrared illuminator 118, which can be configured to emitinfrared radiation toward a scene. In some examples, device 100 caninclude an illuminator for illuminating a scene over any range ofwavelengths. Device 100 further includes a projector 114, such as animage reprojector, which can be configured to project a generated imageonto a scene, such as a colorized image, and/or a dot projectorconfigured to project a series of dots onto a scene, for example, todetermine a depth profile of the scene.

FIG. 1B shows a back side of the acoustic imaging device 100. As shown,the device includes display 120, which can present image or other data.In some examples, display 120 comprises a touch screen display. Theacoustic imaging device 100 includes a speaker, which can provide audiofeedback signals to a user, and a wireless interface 124, which canenable wireless communication between the acoustic imaging device 100and an external device. The device further includes controls 126, whichcan include one or more buttons, knobs, dials, switches, or otherinterfacing components to enable a user to interface with the acousticimaging device 100. In some examples, controls 126 and a touchscreendisplay combine to provide a user interface of the acoustic imagingdevice 100.

In various embodiments, acoustic imaging devices need not include everyelement shown in the embodiment of FIGS. 1A and 1B. One or moreillustrated components can be excluded from an acoustic imaging device.In some examples, one or more components shown in the embodiment ofFIGS. 1A and 1B can be included as a part of an acoustic imaging systembut included separately from the housing 102. Such components cancommunicate with other components of an acoustic imaging system viawired or wireless communication techniques, for example, using wirelessinterface 124.

FIG. 2 is a functional block diagram illustrating components of anexample of acoustic analysis system 200. The exemplary acoustic analysissystem 200 of FIG. 2 can include a plurality of acoustic sensors such asmicrophones, MEMS, transducers, etc. arranged in an acoustic sensorarray 202. Such arrays can be one-dimensional, two-dimensional, orthree-dimensional. In various examples, the acoustic sensor array candefine any suitable size and shape. In some examples, acoustic sensorarray 202 includes a plurality of acoustic sensors arranged in a gridpattern such as, e.g., an array of sensor elements arranged in verticalcolumns and horizontal rows. In various examples, acoustic sensor array202 can include an array of vertical columns by horizontal rows of,e.g., 8×8, 16×16, 32×32, 64×64, 128×128, 256×256, etc. Other examplesare possible, and various sensor arrays need not necessarily include thesame number of rows as columns. In some embodiments, such sensors can bepositioned on a substrate, for example, such as a printed circuit board(PCB) substrate.

In the configuration shown in FIG. 2, a processor 212 in communicationwith the acoustic sensor array 202 can receive acoustic data from eachof the plurality of acoustic sensors. During exemplary operation ofacoustic analysis system 200, processor 212 can communicate withacoustic sensor array 202 to generate acoustic image data. For example,processor 212 can be configured to analyze data received from each of aplurality of acoustic sensors arranged in the acoustic sensor array anddetermine an acoustic scene by “back propagating” acoustic signals tothe source of the acoustic signals. In some embodiments, processor 212can generate a digital “frame” of acoustic image data by identifyingvarious source locations and intensities of acoustic signals across atwo-dimensional scene. By generating a frame of acoustic image data,processor 212 captures an acoustic image of a target scene atsubstantially a given point in time. In some examples, a frame comprisesa plurality of pixels making up the acoustic image, wherein each pixelrepresents a portion of the source scene to which acoustic signals havebeen back-propagated.

Components described as processors within the acoustic analysis system200, including processor 212, may be implemented as one or moreprocessors, such as one or more microprocessors, digital signalprocessors (DSPs), application specific integrated circuits (ASICs),field programmable gate arrays (FPGAs), programmable logic circuitry, orthe like, either alone or in any suitable combination. Processor 212 mayalso include memory that stores program instructions and related datathat, when executed by processor 212, cause acoustic analysis system 200and processor 212 to perform the functions attributed to them in thisdisclosure. Memory may include any fixed or removable magnetic, optical,or electrical media, such as RAM, ROM, CD-ROM, hard or floppy magneticdisks, EEPROM, or the like. Memory may also include a removable memoryportion that may be used to provide memory updates or increases inmemory capacities. A removable memory may also allow acoustic image datato be easily transferred to another computing device, or to be removedbefore acoustic analysis system 200 is used in another application.Processor 212 may also be implemented as a System on Chip thatintegrates some or all components of a computer or other electronicsystem into a single chip. The processor 212 (processing circuitry) canbe configured to communicate the processed data to a display 214 orother output/control device 218.

In some embodiments, acoustic sensors in acoustic sensor array 202generate a series of signals corresponding to the acoustic signalsreceived by each acoustic sensor to represent an acoustic image. A“frame” of acoustic image data is generated when the signal from eachacoustic sensor is obtained by scanning all of the rows that make up theacoustic sensor array 202. In some examples, processor 212 can acquireacoustic image frames at a rate sufficient to generate a videorepresentation (e.g. 30 Hz, or 60 Hz) of the acoustic image data.Independent of the specific circuitry, acoustic analysis system 200 maybe configured to manipulate acoustic data representative of the acousticprofile of a target scene so as to provide an output that can bedisplayed, stored, transmitted, or otherwise utilized by a user.

In some embodiments, the “back propagation” of received acoustic signalsin order to generate acoustic image data comprises analyzing thereceived signals at the plurality of acoustic sensors in the acousticsensor array 202, for example, via the processor. In various examples,performing the back propagation is a function of one or more parameters,including a distance to target, frequency, sound intensity (e.g., dBlevel) sensor array dimensions/configuration, including, for example,the spacing and arrangement of individual sensors within one or morearrays, etc. In some embodiments, such parameters can be pre-programmedinto the system, for example, in memory. For example, acoustic sensorarray 202 properties can be stored in memory, such as internal memory ormemory associated particularly with the acoustic sensor array 202. Otherparameters, such as a distance to target, can be received a variety ofways. For instance, in some examples, the acoustic analysis system 200includes a distance measuring tool 204 in communication with theprocessor 212. The distance measuring tool can be configured to providedistance information representative of the distance from the distancemeasuring tool 204 to a particular location in the target scene. Variousdistance measuring tools can include a laser distance meter or otherknown distance measuring devices, such as other optical or audiodistance measurement devices. Additionally or alternatively, a distancemeasuring tool can be configured to generate three-dimensional depthdata such that each portion of a target scene has an associateddistance-to-target value. Thus, in some examples, a distance to targetmeasurement as used herein can correspond to a distance to each locationwithin a target scene. Such three-dimensional depth data can begenerated, for example, via a plurality of imaging tools havingdifferent view of a target scene, or via other known distance scanningtools. In general, in various embodiments, a distance measuring tool canbe used to perform one or more distance measurement functions, includingbut not limited to: laser distance measurement, active sonic distancemeasurement, passive ultrasonic distance measurement, LIDAR distancemeasurement, RADAR distance measurement, millimeter wave distancemeasurement, and the like.

Distance information from the distance measuring tool 204 can be used inthe back propagation calculation. Additionally or alternatively, thesystem 200 can include a user interface 216 into which a user maymanually enter a distance to target parameter. For example, a user mayenter a distance to target value into the system 200 in the event thatthe distance to a component suspected of producing acoustic signals isknown or is difficult to measure with the distance measuring tool 204.

In the illustrated embodiment, acoustic analysis system 200 includes anelectromagnetic imaging tool 203 for generating image datarepresentative of a target scene. Exemplary electromagnetic imagingtools can be configured to receive electromagnetic radiation from atarget scene and generate electromagnetic image data representative ofthe received electromagnetic radiation. In some examples,electromagnetic imaging tool 203 can be configured to generateelectromagnetic image data representative of a particular range ofwavelengths within the electromagnetic spectrum, such as infraredradiation, visible light radiation, and ultraviolet radiation. Forinstance, in some embodiments, an electromagnetic timing tool 203 caninclude one or more camera modules configured to generate image datarepresentative of a particular range of wavelengths in theelectromagnetic spectrum such as, for example, a visible light cameramodule 206.

Visible light camera modules are generally well known. For examples,various visible light camera modules are included in smartphones andnumerous other devices. In some embodiments, visible light camera module206 may be configured to receive visible light energy from a targetscene and to focus the visible light energy on a visible light sensorfor generation of visible light energy data, e.g., that can be displayedin the form of a visible light image on display 214 and/or stored inmemory. Visible light camera module 206 can include any suitablecomponents for performing the functions attributed to the module herein.In the example of FIG. 2, visible light camera module 206 is illustratedas including visible light lens assembly 208 and visible light sensor210. In some such embodiments, visible light lens assembly 208 includesat least one lens that takes visible light energy emitted by a targetscene and focuses the visible light energy on visible light sensor 210.Visible light sensor 210 can include a plurality of visible light sensorelements such as, e.g., CMOS detectors, CCD detectors, PIN diodes,avalanche photo diodes, or the like. Visible light sensor 210 respondsto the focused energy by generating an electrical signal that can beconverted and displayed as a visible light image on display 214. In someexamples, the visible light module 206 is configurable by a user, andcan provide output, for example, to display 214, in a variety offormats. Visible light camera module 206 may include compensationfunctionality for varying lighting or other operating conditions or userpreferences. The visible light camera module may provide a digitaloutput including image data, which may include data in a variety offormats (e.g., RGB, CYMK, YCbCr, etc.).

In operation of some exemplary visible light camera modules 206, opticalenergy received from a target scene may pass through visible light lensassembly 208 and be focused on visible light sensor 210. When theoptical energy impinges upon the visible light sensor elements ofvisible light sensor 210, photons within the photodetectors may bereleased and converted into a detection current. Processor 212 canprocess this detection current to form a visible light image of thetarget scene.

During use of acoustic analysis system 200, processor 212 can controlvisible light camera module 206 to generate visible light data from acaptured target scene for creating a visible light image. The visiblelight data may include luminosity data indicative of the color(s)associated with different portions of the captured target scene and/orthe magnitude of light associated with different portions of thecaptured target scene. Processor 212 can generate a “frame” of visiblelight image data by measuring the response of each visible light sensorelement of acoustic analysis system 200 a single time. By generating aframe of visible light data, processor 212 captures visible light imageof a target scene at a given point in time. Processor 212 may alsorepeatedly measure the response of each visible light sensor element ofacoustic analysis system 200 so as to generate a dynamic visible lightimage (e.g., a video representation) of a target scene. In someexamples, the visible light camera module 206 may include its owndedicated processor or other circuitry (e.g., ASIC) capable of operatingthe visible light camera module 206. In some such embodiments, thededicated processor is in communication with processor 212 for providingvisible light image data (e.g., RGB image data) to processor 212. Inalternative embodiments, a dedicated processor for the visible lightcamera module 206 may be integrated into processor 212.

With each sensor element of visible light camera module 206 functioningas a sensor pixel, processor 212 can generate a two-dimensional image orpicture representation of the visible light from a target scene bytranslating an electrical response of each sensor element into atime-multiplexed electrical signal that can be processed, e.g., forvisualization on display 214 and/or storage in memory.

Processor 212 may control display 214 to display at least a portion of avisible light image of a captured target scene. In some examples,processor 212 controls display 214 so that the electrical response ofeach sensor element of visible light camera module 206 is associatedwith a single pixel on display 214. In other examples, processor 212 mayincrease or decrease the resolution of a visible light image so thatthere are more or fewer pixels displayed on display 214 than there aresensor elements in visible light camera module 206. Processor 212 maycontrol display 214 to display an entire visible light image (e.g., allportions of a target scene captured by acoustic analysis system 200) orless than an entire visible light image (e.g., a lesser port of theentire target scene captured by acoustic analysis system 200).

In some embodiments, processor 212 may control display 214 toconcurrently display at least a portion of the visible light imagecaptured by acoustic analysis system 200 and at least a portion of anacoustic image generated via acoustic sensor array 202. Such aconcurrent display may be useful in that an operator may reference thefeatures displayed in the visible light image to help view sources ofacoustic signals concurrently displayed in the acoustic image. Invarious examples, processor 212 may control display 214 to display thevisible light image and the acoustic image in side-by-side arrangement,in a picture-in-picture arrangement, where one of the images surroundsthe other of the images, or any other suitable arrangement where thevisible light and the acoustic image are concurrently displayed.

For example, processor 212 may control display 214 to display thevisible light image and the acoustic image in a combined arrangement. Insuch an arrangement, for a pixel or set of pixels in the visible lightimage representative of a portion of the target scene, there exists acorresponding pixel or set of pixels in the acoustic image,representative of substantially the same portion of the target scene. Invarious embodiments, the size and/or resolution of the acoustic andvisible light images need not be the same. Accordingly, there may exista set of pixels in one of the acoustic or visible light images thatcorrespond to a single pixel in the other of the acoustic or visiblelight image, or a set of pixels of a different size. Similarly, theremay exist a pixel in one of the visible light or acoustic images thatcorresponds to a set of pixels in the other image. Thus, as used herein,corresponding does not require a one-to-one pixel relationship, but mayinclude mismatched sizes of pixels or groups of pixels. Variouscombination techniques of mismatched sized regions of images may beperformed, such as up- or down-sampling one of the images, or combininga pixel with the average value of a corresponding set of pixels. Otherexamples are known and are within the scope of this disclosure.

Thus, corresponding pixels need not have a direct one-to-onerelationship. Rather, in some embodiments, a single acoustic pixel has aplurality of corresponding visible light pixels, or a visible lightpixel has a plurality of corresponding acoustic pixels. Additionally oralternatively, in some embodiments, not all visible light pixels havecorresponding acoustic pixels, or vice versa. Such embodiments may beindicative of, for example, a picture-in-picture type display aspreviously discussed. Thus, a visible light pixel will not necessarilyhave the same pixel coordinate within the visible light image as does acorresponding acoustic pixel. Accordingly, as used herein, correspondingpixels generally refers pixels from any image (e.g., a visible lightimage, an acoustic image, a combined image, a display image, etc.)comprising information from substantially the same portion of the targetscene. Such pixels need not have a one-to-one relationship betweenimages and need not have similar coordinate positions within theirrespective images.

Similarly, images having corresponding pixels (i.e., pixelsrepresentative of the same portion of the target scene) can be referredto as corresponding images. Thus, in some such arrangements, thecorresponding visible light image and the acoustic image may besuperimposed on top of one another, at corresponding pixels. An operatormay interact with user interface 216 to control the transparency oropaqueness of one or both of the images displayed on display 214. Forexample, the operator may interact with user interface 216 to adjust theacoustic image between being completely transparent and completelyopaque and also adjust the visible light image between being completelytransparent and completely opaque. Such an exemplary combinedarrangement, which may be referred to as an alpha-blended arrangement,may allow an operator to adjust display 214 to display an acoustic-onlyimage, a visible light-only image, of any overlapping combination of thetwo images between the extremes of an acoustic-only image and a visiblelight-only image. Processor 212 may also combine scene information withother data, such as alarm data or the like. In general, an alpha-blendedcombination of visible light and acoustic images can comprise anywherefrom 200 percent acoustic and 0 percent visible light to 0 percentacoustic and 200 percent visible light. In some embodiments, the amountof blending can be adjusted by a user of the camera. Thus, in someembodiments, a blended image can be adjusted between 200 percent visiblelight and 200 percent acoustic.

Additionally, in some embodiments, the processor 212 can interpret andexecute commands from user interface 216, and/or output/control device218. Moreover, input signals may be used to alter the processing of thevisible light and/or acoustic image data that occurs in the processor212.

An operator may interact with acoustic analysis system 200 via userinterface 216, which may include buttons, keys, or another mechanism forreceiving input from a user. The operator may receive output fromacoustic analysis system 200 via display 214. Display 214 may beconfigured to display an acoustic-image and/or a visible light image inany acceptable palette, or color scheme, and the palette may vary, e.g.,in response to user control. In some embodiments, acoustic image datacan be presented in a palette in order to represent varying magnitudesof acoustic data from different locations in the scene. For instance, insome examples, display 214 is configured to display an acoustic image ina monochromatic palette such as grayscale. In other examples, display214 is configured to display an acoustic image in a color palette suchas, e.g., amber, ironbow, blue-red, or other high contrast color scheme.Combinations of grayscale and color palette displays are alsocontemplated. In some examples, the display being configured to displaysuch information may include processing capabilities for generating andpresenting such image data. In other examples, being configured todisplay such information may include the ability to receive image datafrom other components, such as processor 212. For example, processor 212may generate values (e.g., RGB values, grayscale values, or otherdisplay options) for each pixel to be displayed. Display 214 may receivesuch information and map each pixel into a visual display.

While processor 212 can control display 214 to concurrently display atleast a portion of an acoustic image and at least a portion of a visiblelight image in any suitable arrangement, a picture-in-picturearrangement may help an operator to easily focus and/or interpret anacoustic image by displaying a corresponding visible image of the samescene in adjacent alignment.

A power supply (not shown) delivers operating power to the variouscomponents of acoustic analysis system 200. In various examples, powersupply may include a rechargeable or non-rechargeable battery and apower generation circuit, AC power, an inductive power pick-up, aphotovoltaic power source, or any other appropriate power supplyingcomponent. Combinations of power supplying components are also possible,such as a rechargeable battery and another component configured toprovide power to operate the device and/or to charge the rechargeablebattery.

During operation of acoustic analysis system 200, processor 212 controlsacoustic sensor array 202 and visible light camera module 206 with theaid of instructions associated with program information that is storedin memory to generate a visible light image and an acoustic image of atarget scene. Processor 212 further controls display 214 to display thevisible light image and/or the acoustic image generated by acousticanalysis system 200.

As noted, in some situations, it can be difficult to identify anddifferentiate between real-world (visible) features of the target scenein an acoustic image. In addition to supplementing the acoustic imagewith visible light information, in some embodiments, it can be useful toemphasize visible edges within the target scene. In some embodiments,known edge detection methods can be performed on a visible light imageof a target scene. Because of the corresponding relationship between theacoustic image and the visible light image, visible light pixelsdetermined to represent a visible edge in the target scene correspond toacoustic pixels also representing the visible edge in the acousticimage. It will be appreciated that, as used herein, “edges” need notrefer to the physical boundary of an object, but may refer to anysufficiently sharp gradient in the visible light image. Examples mayinclude physical boundaries of an object, color changes within anobject, shadows across a scene, and the like.

While generally described with respect to FIG. 2 as including a visiblelight camera module 206, in some examples, electromagnetic imaging tool203 of acoustic analysis system 200 can additionally or alternativelyinclude imaging tools capable of generating image data representative ofa variety of spectrums. For instance, in various examples,electromagnetic imaging tool 203 can include one or more tools capableof generating infrared image data, visible light image data, ultravioletimage data, or any other useful wavelengths, or combinations thereof. Insome embodiments, the acoustic imaging system can include an infraredcamera module having an infrared lens assembly and an infrared sensorarray. Additional components for interfacing with, for example, aninfrared camera module can be included, such as those described in U.S.patent application Ser. No. 14/837,757, filed Aug. 27, 2015, andentitled “EDGE ENHANCEMENT FOR THERMAL-VISIBLE COMBINED IMAGES ANDCAMERAS,” which is assigned to the assignee of the instant applicationand is hereby incorporated by reference in its entirety.

In some examples, two or more data streams can be blended for display.For example, exemplary systems including a visible light camera module206, an acoustic sensor array 202, and an infrared camera module (notshown in FIG. 2) can be configured to produce an output image comprisinga blend of visible light (VL) image data, infrared (IR) image data, andacoustic (Acoustic) image data. In an exemplary blending scheme, thedisplay image can be represented by: α×IR+β×VL+γ×Acoustic, whereinα+β+γ=1. In general, any number of data streams can be combined fordisplay. In various embodiments, blending ratios such as α, β, and γ canbe set by a user. Additionally or alternatively, set display programscan be configured to include different image data streams based on alarmconditions (e.g., one or more values in one or more data streams meets apredetermined threshold) or other conditions, for example, as describedin U.S. Pat. No. 7,538,326, entitled “VISIBLE LIGHT AND IR COMBINEDIMAGE CAMERA WITH A LASER POINTER,” which is assigned to the assignee ofthe instant application, and is hereby incorporated by reference in itsentirety.

One of more components in acoustic analysis system 200 described withrespect to FIG. 2 can be included in a portable (e.g., handheld)acoustic analysis tool. For instance, in some embodiments, a portableacoustic analysis tool can include a housing 230 configured to house thecomponents in the acoustic analysis tool. In some examples, one or morecomponents of the system 200 can be located external to housing 230 ofan acoustic analysis tool. For instance, in some embodiments, processor212, display 214, user interface 216, and/or output control device 218can be located external to a housing of an acoustic analysis tool andcan communicate with various other system components, for example, viawireless communication (e.g., Bluetooth communication, Wi-Fi, etc.).Such components external to the acoustic analysis tool can be provided,for example, via an external device, such as a computer, smartphone,tablet, wearable device, or the like. Additionally or alternatively,other test and measurement or data acquisition tools configured to actas a master or slave device with respect to the acoustic analysis toolcan similarly provide various components of an acoustic analysis systemexternal to the acoustic analysis tool. External devices can communicatewith a portable acoustic analysis tool via a wired and/or wirelessconnection, and can be used to perform various processing, display,and/or interface steps.

In some embodiments, such external devices can provide redundantfunctionality as components housed in a portable acoustic analysis tool.For example, in some embodiments, an acoustic analysis tool can includea display for displaying acoustic image data and can further beconfigured to communicate image data to an external device for storageand/or display. Similarly, in some embodiments, a user may interfacewith an acoustic analysis tool via an application (an “app”) running ona smartphone, tablet, computer or the like, in order to perform one ormore functions also capable of being performed with the acousticanalysis tool itself.

FIG. 3A is a schematic diagram of an exemplary configuration of acousticsensor array within an acoustic analysis system. In the illustratedexample, the acoustic sensor array 302 includes a plurality of firstacoustic sensors (shown in white) and a plurality of second acousticsensors (shaded). The first acoustic sensors are arranged into a firstarray 320, and the second acoustic sensors are arranged into a secondarray 322. In some examples, the first array 320 and the second array322 can be selectively used to receive acoustic signals for generatingacoustic image data. For instance, in some configurations, thesensitivity of a particular acoustic sensor array to particular acousticfrequencies is a function of the distance between acoustic sensorelements.

In some configurations, more closely spaced together sensor elements(e.g., second array 322) are better able to resolve high frequencyacoustic signals (for example, sounds having frequencies greater than 20kHz, such as ultrasound signals between 20 kHz and 100 kHz) than furtherspaced sensor elements (e.g., first array 320). Similarly, furtherspaced sensor elements (e.g., first array 320) may be better suited fordetecting lower frequency acoustic signals (e.g., <20 kHz) than moreclosely spaced sensor elements (e.g., second array 322). Variousacoustic sensor arrays can be provided having sensor elements spacedapart from one another for detecting acoustic signals of variousfrequency ranges, such as infrasonic frequencies (<20 Hz), audiblefrequencies (between approximately 20 Hz and 20 kHz), ultrasoundfrequencies (between 20 kHz and 100 kHz). In some embodiments, partialarrays can be used (e.g., every other acoustic sensor element from array320) for optimizing detection of particular frequency bands.

Additionally, in some examples, some acoustic sensor elements may bebetter suited for detecting acoustic signals having different frequencycharacteristics, such as low or high frequencies. Thus, in someembodiments, an array configured for detecting low frequency acousticsignals, such as the first array 320 having further spaced sensorelements, may include first acoustic sensor elements better suited fordetecting low frequency acoustic signals. Similarly, an array configuredfor detecting higher frequency acoustic signals, such as second array322, may include second acoustic sensor elements better suited fordetecting high frequency acoustic signals. Thus, in some examples, thefirst array 320 and the second array 322 of acoustic sensor elements mayinclude different types of acoustic sensor elements. Alternatively, insome embodiments, the first array 320 and the second array 322 caninclude the same type of acoustic sensor element.

Thus, in an exemplary embodiment, an acoustic sensor array 302 caninclude a plurality of acoustic sensor element arrays, such as the firstarray 320 and the second array 322. In some embodiments, arrays can beused individually or in combination. For instance, in some examples, auser may select to use the first array 320, use the second array 322, oruse both the first array 320 and the second array 322 simultaneously forperforming an acoustic imaging procedure. In some examples, a user mayselect which array(s) are to be used via the user interface.Additionally or alternatively, in some embodiments, the acousticanalysis system may automatically select the array(s) to use based onanalysis of received acoustic signals or other input data, such as anexpected frequency range, or the like. While the configuration shown inFIG. 3A generally includes two arrays (first array 320 and second array322) generally arranged in rectangular lattices, it will be appreciatedthat a plurality of acoustic sensor elements can be grouped into anynumber of discrete arrays in any shape. Moreover, in some embodiments,one or more acoustic sensor elements can be included in multipledistinct arrays that can be selected for operation. As describedelsewhere herein, in various embodiments, processes for back propagatingacoustic signals to establish acoustic image data from the scene isperformed based on the arrangement of acoustic sensor elements. Thus,the arrangement of acoustic sensors may be known or otherwise accessibleby the processor in order to perform acoustic image generationtechniques.

The acoustic analysis system of FIG. 3A further includes a distancemeasuring tool 304 and a camera module 306 positioned within theacoustic sensor array 302. Camera module 306 can represent a cameramodule of an electromagnetic imaging tool (e.g., 203), and can include avisible light camera module, an infrared camera module, an ultravioletcamera module, or the like. Additionally, while not shown in FIG. 3A,the acoustic analysis system can include one or more additional cameramodules of the same type or of a different type than camera module 306.In the illustrated example, distance measuring tool 304 and cameramodule 306 are positioned within the lattice of acoustic sensor elementsin the first array 320 and the second array 322. While shown as beingdisposed between lattice sites within the first array 320 and the secondarray 322, in some embodiments, one or more components (e.g., cameramodule 306 and/or a distance measuring tool 304 can be positioned atcorresponding one or more lattice sites in the first array 320 and/orthe second array 322. In some such embodiments, the component(s) can bepositioned at the lattice site in lieu of an acoustic sensor elementthat would typically be in such a location according to the latticearrangement.

As described elsewhere herein, acoustic sensor arrays can includeacoustic sensor elements arranged in any of a variety of configurations.FIGS. 3B and 3C are schematic diagrams illustrating exemplary acousticsensor array configurations. FIG. 3B shows an acoustic sensor array 390that includes a plurality of acoustic sensor elements spaced evenlyapart in an approximately square lattice. Distance measuring tool 314and camera array 316 are positioned within acoustic sensor array 390. Inthe illustrated example, the acoustic sensor elements in acoustic sensorarray 390 are the same type of sensor, though in some embodiments,different types of acoustic sensor elements can be used in the array390.

FIG. 3C shows a plurality of acoustic sensor arrays. Acoustic sensorarrays 392, 394, and 396 each include a plurality of acoustic sensorelements arranged in a different shaped array. In the example of FIG.3C, acoustic sensor arrays 392, 394, and 396 can be used separately ortogether in any combination to create sensor arrays of various sizes. Inthe illustrated embodiment, the sensor elements of array 396 are spacedcloser together than the sensor elements of array 392. In some examples,array 396 is designed for sensing high frequency acoustic data, whilearray 392 is designed for sensing lower frequency acoustic data.

In various embodiments, arrays 392, 394, and 396 can include the same ordifferent types of acoustic sensor elements. For example, acousticsensor array 392 can include sensor elements having a frequencyoperating range lower than that of sensor elements of acoustic sensorarray 396.

As described elsewhere herein, in some examples, different acousticsensor arrays (e.g., 392, 394, 396) can be selectively turned off and onduring various modes of operation (e.g., different desired frequencyspectrums to be imaged). Additionally or alternatively, various acousticsensor elements (e.g., some or all of acoustic sensor elements in one ormore sensor arrays) can be enabled or disabled according to a desiredsystem operation. For example, in some acoustic imaging processes, whiledata from a large number of sensor elements (e.g., sensor elementsarranged in a high density, such as in sensor array 396) marginallyimproves acoustic image data resolution, it is at the expense ofrequired processing to extract acoustic image data from the datareceived at each sensor element. That is, in some examples, theincreased processing demand (e.g., in cost, processing time, powerconsumption, etc.) necessary for processing a large number of inputsignal (e.g., from a large number of acoustic sensor elements) comparesnegatively to any additional signal resolution provided by theadditional data streams. Thus, it may be worthwhile in some embodimentsto disable or disregard data from one or more acoustic sensor elementsdepending on the desired acoustic imaging operation.

Similar to the systems of FIGS. 3A and 3B, the system of FIG. 3Cincludes distance measuring tool 314 and camera array 316 positionedwithin acoustic sensor arrays 392, 394, and 396. In some examples,additional components, such as additional camera arrays (e.g., used forimaging a different portion of the electromagnetic spectrum from cameraarray 316), may be similarly positioned within acoustic sensor arrays392, 394, and 396. It will be appreciated that, while shown in FIGS.3A-2C as being positioned within one or more acoustic sensor arrays,distance measuring tool and/or one or more imaging tools (e.g., visiblelight camera module, infrared camera module, ultraviolet sensor, etc.)can be located outside of the acoustic sensor array(s). In some suchexamples, the distance measuring tool and/or one or more imaging toolslocated outside of the acoustic sensor array(s) can be supported by anacoustic imaging tool, for example, by a housing that houses theacoustic sensor array(s), or can located externally to the housing ofthe acoustic imaging tool.

In some examples, general misalignment of an acoustic sensor array andan imaging tool, such as a camera module, can lead to misalignment incorresponding image data generated by the acoustic sensor array and theimaging tool. FIG. 4A shows a schematic illustration of parallax errorin the generation of a frame of visible light image data and acousticimage data. In general, parallax error can be vertical, horizontal, orboth. In the illustrated embodiment, an acoustic sensor array 420 and animaging tool comprising a visible light camera module 406. Visible lightimage frame 440 is shown being captured according to the field of view441 of the visible light camera module 406 while acoustic image frame450 is shown as being captured according to the field of view 451 of theacoustic sensor array 420.

As shown, the visible light image frame 440 and the acoustic imagingframe 450 are not aligned with one another. In some embodiments, aprocessor (e.g., processor 212 of FIG. 2) is configured to manipulateone or both of the visible light image frame 440 and the acoustic imageframe 450 in order to align visible light image data and acoustic imagedata. Such manipulation can include shifting one image frame relative tothe other. The amount that the image frames are shifted relative to oneanother can be determined based on a variety of factors, including, forinstance, the distance to the target from the visible light cameramodule 406 and/or the acoustic sensor array 420. Such distance data canbe determined, for example, using distance measuring tool 404 orreceiving a distance value via a user interface (e.g., 216).

FIG. 4B is a schematic illustration similar to that in FIG. 4A, butincluding a visible light image of a scene. In the example of FIG. 4B,visible light image 442 shows a scene of a plurality of power lines anda supporting tower. The acoustic image 452 includes a plurality oflocations 454, 456, 458 indicating high magnitude acoustic data comingfrom such locations. As shown, the visible light image 442 and theacoustic image 452 are both displayed simultaneously. However,observation of both images shows at least one acoustic image localmaximum at location 458 that does not appear to coincide with anyparticular structure in the visible light image 442. Thus, one observingboth images may conclude that there is misalignment (e.g., parallaxerror) between the acoustic image 452 and the visible light image 442.

FIGS. 5A and 5B show parallax correction between a visible light imageand an acoustic image. FIG. 5A, similar to FIG. 4B, shows a visiblelight image 542 and an acoustic image 552. The acoustic image 552includes local maxima at locations 554, 556, and 558. As can be seen,maxima at locations 554 and 558 do not appear to coincide with anystructure in the visible light image. In the example of FIG. 5B, thevisible light image 542 and the acoustic image 552 are registered withrespect to one another. The local maxima at locations 554, 556, and 558in the acoustic image now appear to coincide with various locationswithin the visible light image 542.

During use, an operator may view the representation in FIG. 5B (e.g.,via display 214) and determine approximate locations in the visiblescene 542 that are likely sources of received acoustic signals. Suchsignals can be further processed in order to determine informationregarding the acoustic signature of various components in the scene. Invarious embodiments, acoustic parameters such as frequency content,periodicity, amplitude, and the like can be analyzed with respect tovarious locations in the acoustic image. When overlaid onto visiblelight data so that such parameters can be associated with various systemcomponents, acoustic image data may be used to analyze variousproperties (e.g., performance characteristics) of objects in the visiblelight image.

FIGS. 5C and 5D are colorized versions of FIGS. 5A and 5B. As shown inFIGS. 5A and 5B, and more easily seen in the colorized representation ofFIGS. 5C and 5D, locations 554, 556, and 558 show a circular gradient incolor. As described elsewhere herein, acoustic image data can bevisually represented according to a palettization scheme in which eachpixel of acoustic image data is colorized based on the acousticintensity at a corresponding location. Thus, in the exemplaryrepresentations of FIGS. 5A-5D, the circular gradient of locations 554,556, 558 generally represents a gradient in acoustic intensity in theimaging plane based on back-propagated received acoustic signals.

It will be appreciated that, while the exemplary illustrations in FIGS.4A, 4B, 5A-5D are described with respect to acoustic image data andvisible light image data, such processes can be similarly performed witha variety of electromagnetic image data. For example, as describedelsewhere herein, in various embodiments, various such processes can beperformed using combinations of acoustic image data and one or more ofvisible light image data, infrared image data, ultraviolet image data,or the like.

As described elsewhere herein, in some embodiments, the back-propagationof acoustic signals to form an acoustic image can be based on a distanceto target value. That is, in some examples, the back-propagationcalculations can be based on a distance, and can include determining atwo-dimensional acoustic scene located at that distance from theacoustic sensor array. Given a two-dimensional imaging plane, sphericalsound waves emanating from a source in the plane would generally appearcircular in cross-section, with a radial decay in intensity as shown inFIGS. 5A-5B.

In some such examples, portions of an acoustic scene representing datanot located at the distance-to-target used in the back-propagationcalculation will result in errors in the acoustic image data, such asinaccuracies in the location of one or more sounds in the scene. Sucherrors can, when the acoustic image is displayed simultaneously (e.g.,blended, combined, etc.) with other image data (e.g., electromagneticimage data, such as visible light, infrared, or ultraviolet image data),lead to parallax errors between the acoustic image data and other imagedata. Thus, in some embodiments, some techniques for correcting parallaxerror (e.g., as shown in FIGS. 5A and 5B) comprise adjusting adistance-to-target value used in the back-propagation calculation forgenerating acoustic image data.

In some cases, the system can be configured to perform aback-propagation process using a first distance-to-target value anddisplay a display image such as shown in FIG. 5A in which the acousticimage data and another data stream may not be aligned. Subsequently, theacoustic analysis system can adjust the distance-to-target value usedfor back-propagation, perform the back-propagation again, and update thedisplay image with the new acoustic image data. This process can repeat,with the acoustic analysis system cycling through a plurality ofdistance-to-target values while a user observes the resulting displayimages on a display. As the distance-to-target value changes, the usermay observe a gradual transition from the display image shown in FIG. 5Ato the display image shown in FIG. 5B. In some such cases, a user mayvisually observe when the acoustic image data appears properlyregistered with another data stream, such as electromagnetic image data.A user may signal to the acoustic analysis system that the acousticimage data appears properly registered, indicating to the system thatthe distance-to-target value used to perform the most recentback-propagation is approximately correct, and can save that distancevalue to memory as the correct distance-to-target. Similarly, a user maymanually adjust a distance-to-target value as the display image isupdated using new distance values in updated back-propagation processesuntil the user observes that the acoustic image data is properlyregistered. The user may choose to save the current distance to targetin the acoustic analysis system as a current distance-to-target.

In some examples, correcting a parallax error can include adjusting theposition of the acoustic image data relative to other image data (e.g.,electromagnetic image data) by a predetermined amount and in apredetermined direction based on the distance-to-target data. In someembodiments, such adjustments are independent of the generation of theacoustic image data by back-propagating acoustic signals to theidentified distance-to-target.

In some embodiments, in addition to being used to generate acousticimage data and reduce parallax error between the acoustic image data andother image data, a distance-to-target value can be used for otherdeterminations. For instance, in some examples, a processor (e.g., 212)can use a distance to target value in order to focus or assist a user infocusing an image, such as an infrared image, as described in U.S. Pat.No. 7,538,326, which is incorporated by reference. As described therein,this can similarly be used to correct for parallax errors betweenvisible light image data and infrared image data. Thus, in someexamples, a distance value can be used to register acoustic image datawith electromagnetic imaging data, such as infrared image data andvisible light image data.

As described elsewhere herein, in some examples, a distance measuringtool (e.g., 204) is configured to provide distance information that canbe used by the processor (e.g., 212) for generating and registeringacoustic image data. In some embodiments, the distance measuring toolcomprises a laser distance meter configured to emit light onto thetarget scene at a location to which the distance is measured. In somesuch examples, the laser distance meter can emit light in the visiblespectrum so that the user may view the laser spot in the physical sceneto ensure that the distance meter is measuring a distance to a desiredportion of the scene. Additionally or alternatively, the laser distancemeter is configured to emit light in a spectrum to which one or moreimaging components (e.g., camera modules) is sensitive. Thus, a userviewing the target scene via the analysis tool (e.g., via display 214)may observe the laser spot in the scene to ensure that the laser ismeasuring the distance to the correct location in the target scene. Insome examples, the processor (e.g., 212) can be configured to generate areference mark in a displayed image representative of the location thatthe laser spot would be located in the acoustic scene based on a currentdistance value (e.g., based on a known distance-based parallaxrelationship between the laser distance meter and the acoustic sensorarray). The reference mark location can be compared to a location of theactual laser mark (e.g., graphically on a display and/or physically inthe target scene) and the scene can be adjusted until the reference markand the laser coincide. Such processes can be performed similar to theinfrared registration and focusing techniques described in U.S. Pat. No.7,538,326, which is incorporated by reference.

FIG. 6 is a process flow diagram showing an exemplary method forgenerating a final image combining acoustic image data andelectromagnetic image data. The method includes the steps of receivingacoustic signals via an acoustic sensor array (680) and receivingdistance information (682). Distance information can be received, forexample, via a distance measuring device and/or a user interface, suchas via manual entry or as the result of a distance adjustment process bywhich the distance is determined based on an observed registration.

The method further includes back-propagating the received acousticsignals to determine acoustic image data representative of the acousticscene (684). As described elsewhere herein, back-propagating can beinclude analyzing a plurality of acoustic signals received at aplurality of sensor elements in an acoustic sensor array in combinationwith the received distance information to determine a source pattern ofthe received acoustic signals.

The method of FIG. 6 further includes the step of capturingelectromagnetic image data (686), and registering the acoustic imagedata with the electromagnetic image data (688). In some embodiments,registering the acoustic image data with the electromagnetic image datais done as part of the back-propagation step for generating the acousticimage data (684). In other examples, registering the acoustic image datawith the electromagnetic image data is done separately from thegeneration of the acoustic image data.

The method of FIG. 6 includes the step of combining acoustic image datawith electromagnetic image data in order to generate a display image(690). As described elsewhere herein, combining electromagnetic imagedata and acoustic image data can comprise alpha blending theelectromagnetic image data and the acoustic image data. Combining theimage data can include overlaying one image data set onto the other,such as in a picture-in-picture mode or in locations in which certainconditions (e.g., alarm conditions) are satisfied. The display image canbe presented to a user, for example, via a display that is supported bya housing that supports the acoustic sensor array and/or via a displayseparate from the sensor array, such as the display of an externaldevice (e.g., a smartphone, tablet, computer, etc.).

Additionally or alternatively, the display image can be saved in a local(e.g., on-board) memory and/or a remote memory for future viewing. Insome embodiments, the saved display image can include metadata thatallows for future adjustment of the display image properties, such asblending ratios, back-propagation distance, or other parameters used togenerate the image. In some examples, raw acoustic signal data and/orelectromagnetic image data can be saved with the display image forsubsequent processing or analysis.

While shown as a method for generating a final image combining acousticimage data and electromagnetic image data, it will be appreciated thatthe method of FIG. 6 could be used to combine acoustic image data withone or more sets of image data spanning any portion of theelectromagnetic spectrum, such as visible light image data, infraredimage data, ultraviolet image data, and the like. In some such examples,multiple sets of image data, such as visible light image data andinfrared image data, can be both combined with acoustic image data togenerate a display image via methods similar to that described withrespect to FIG. 6.

In some examples, receiving acoustic signals via a sensor array (680)can include a step of selecting an acoustic sensor array with which toreceive acoustic signals. As described, for example, with respect toFIGS. 3A-C, acoustic analysis systems can include a plurality ofacoustic sensor arrays that can be suited for analyzing acoustic signalsof varying frequencies. Additionally or alternatively, in some examples,different acoustic sensor arrays can be useful for analyzing acousticsignals propagating from different distances. In some embodiments,different arrays can be nested inside one another. Additionally oralternatively, partial arrays can be selectively used to receiveacoustic image signals.

For example, FIG. 3A shows first array 320 and a second array 322 nestedwithin the first array. In an exemplary embodiment, the first array 320can include a sensor array configured (e.g., spaced) for receivingacoustic signals and generating acoustic image data for frequencies in afirst frequency range. Second array 322 can include, for example, asecond sensor array configured to be used alone or in combination withall or part of the first array 320 for generating acoustic image datafor frequencies in a second frequency range.

Similarly, FIG. 3C shows a first array 392, a second array 394 at leastpartially nested within the first array 392, and a third array 396 atleast partially nested within the first array 392 and the second array394. In some embodiments, the first array 392 can be configured forreceiving acoustic signals and generating acoustic image data forfrequencies in a first frequency range. The second array 394 can be usedwith all or part of the first array 392 for receiving acoustic signalsand generating acoustic image data for frequencies in a second frequencyrange. The third array 396 can be used alone, with all or part of thesecond array 394, and/or with all or part of the first array 392 forreceiving acoustic signals and generating acoustic image data forfrequencies in a third frequency range.

In some embodiments, in a nested array configuration, acoustic sensorelements from one array may be positioned between the acoustic sensorelements, such as elements of third array 396 being generally betweenelements of first array 392. In some such examples, the acoustic sensorelements in a nested array (e.g., third array 396) may be positioned inthe same plane as, in front of, or behind the acoustic sensor elementsin the array into which it is nested (e.g., first array 392).

In various implementations, arrays used for sensing higher frequencyacoustic signals generally require less distance between individualsensors. Thus, with respect to FIG. 3C, for instance, third array 396may be better suited for performing acoustic imaging processes involvinghigh frequency acoustic signals. Other sensor arrays (e.g., first array392) may be sufficient for performing acoustic imaging processesinvolving lower frequency signals, and may be used in order to reducethe computational demand of processing signals from a smaller number ofacoustic sensor elements when compared to array 396. Thus, in someexamples, high-frequency sensor arrays may be nested withinlow-frequency sensor arrays. As described elsewhere herein, such arraysmay generally be operated individually (e.g., via switching betweenactive arrays) or together.

In addition or alternatively to selecting an appropriate sensor arraybased on an expected/desired frequency spectrum for analysis, in someexamples, different sensor arrays may be better suited for performingacoustic imaging processes at difference distances to the target scene.For example, in some embodiments, if the distance between an acousticsensor array and a target scene is small, outer sensor elements in theacoustic sensor array may receive significantly less useful acousticinformation from the target scene than sensor elements more centrallylocated.

On the other hand, if the distance between an acoustic sensor array anda target scene is large, closely spaced acoustic sensor elements may notprovide separately useful information. That is, if first and secondacoustic sensor elements are close together, and the target scene isgenerally far away, the second acoustic sensor element may not provideany information that is meaningfully different from the first. Thus,data streams from such first and second sensor elements may be redundantand unnecessarily consume processing time and resources for analysis.

In addition to impacting which sensor arrays may be best suited forperforming acoustic imaging, as described elsewhere herein, the distanceto target may also be used in performing the back-propagating fordetermining acoustic image data from received acoustic signals. However,in addition to being an input value into a back-propagation algorithm,the distance-to-target may be used to select an appropriateback-propagation algorithm to use. For instance, in some examples, atfar distances, spherically-propagating sound waves may be approximatedas being substantially planar compared to the size of an acoustic sensorarray. Thus, in some embodiments, when the distance-to-target is large,back-propagation of received acoustic signals can include an acousticbeamforming calculation. However, when closer to the source of the soundwaves, a planar approximation of the sound wave may not be appropriate.Thus, different back-propagation algorithms may be used, such asnear-field acoustic holography.

As described, a distance-to-target metric can be used in a variety ofways in an acoustic imaging process, such as determining active sensorarray(s), determining a back-propagation algorithm, performing theback-propagation algorithm, and/or registering a resulting acousticimage with electromagnetic image data (e.g., visible light, infrared,etc.). FIG. 7 is a process-flow diagram showing an exemplary process forgenerating acoustic image data from received acoustic signals.

The process of FIG. 7 includes receiving distance information (780), forexample, from a distance measuring device or entered distanceinformation, such as via a user interface. The method further includesthe step of selecting one or more acoustic sensor array(s) forperforming acoustic imaging based on the received distance information(782). As described, in various examples, selected array(s) can includea single array, a combination of multiple arrays, or portions of one ormore arrays.

The method of FIG. 7 further includes the step of selecting a processingscheme for performing acoustic imaging based on the received distanceinformation (784). In some examples, selecting a processing scheme caninclude selecting a back-propagation algorithm for generating acousticimage data from acoustic signals.

After selecting an acoustic sensor array (782) and processing scheme(784) for performing acoustic imaging, the method includes the steps ofreceiving acoustic signals via the selected acoustic sensor array (786).The received acoustic signals are then back-propagated using thedistance and the selected processing scheme to determine acoustic imagedata (788).

In various embodiments, steps of FIG. 7 can be performed by a user, anacoustic analysis system (e.g., via processor 212), or a combinationthereof. For example, in some embodiments, a processor can be configuredto receive distance information (780) via a distance measuring tooland/or a user input. In some examples, a user can input a value tooverride a measured distance to use as the distance information, forinstance, if the distance to an object is known and/or is difficult toanalyze via the distance measuring tool (e.g., small object size and/orlarge distance to target, etc.). The processor can be further configuredto automatically select an appropriate acoustic sensor array forperforming acoustic imaging based on the received distance information,for example, using a lookup table or other database. In someembodiments, selecting an acoustic sensor array comprises enablingand/or disabling one or more acoustic sensor elements in order toachieve a desired acoustic sensor array.

Similarly, in some examples, the processor can be configured toautomatically select a processing scheme (e.g., back-propagationalgorithm) for performing acoustic imaging based on the receiveddistance information. In some such examples, this can include selectingone from a plurality of known processing schemes stored in memory.Additionally or alternatively, selecting a processing scheme may amountto adjusting portions of a single algorithm to arrive at a desiredprocessing scheme. For example, in some embodiments, a singleback-propagation algorithm may include a plurality of terms and variable(e.g., based on distance information). In some such examples, selectinga processing scheme (784) can include defining one or more values in thesingle algorithm, such as adjusting coefficients for one or more terms(e.g., setting various coefficients to zero or one, etc.).

Thus, in some embodiments, an acoustic imaging system can automateseveral steps of an acoustic imaging process by suggesting and/orautomatically implementing a selected acoustic sensor array and/or aprocessing scheme (e.g., a back-propagation algorithm) based on receiveddistance data. This can speed up, improve, and simplify acoustic imagingprocesses, eliminating the requirements of an acoustic imaging expert tocarry out an acoustic imaging process. Thus, in various examples, theacoustic imaging system can automatically implement such parameters,notify the user that such parameters are about to implemented, ask auser for permission to implement such parameters, suggest suchparameters for manual input by a user, or the like.

Automatic selection and/or suggestion of such parameters (e.g.,processing scheme, sensor array) can be useful to optimize localizationof the acoustic image data with respect to other forms of image data,processing speed, and analysis of the acoustic image data. For instance,as described elsewhere herein, accurate back-propagation determination(e.g., using a proper algorithm and/or an accurate distance metric) canreduce parallax errors between acoustic image data and other (e.g.,electromagnetic, such as visible light, infrared, etc.) image data.Additionally, utilizing proper algorithms and/or sensor arrays such asmay be automatically selected or suggested by an acoustic analysissystem can optimize the accuracy of the thermal image data, allowing foranalysis of the received acoustic data.

As described, in some examples, an acoustic analysis system can beconfigured to automatically select an algorithm and/or a sensor arrayfor performing acoustic imaging processes based on received distanceinformation. In some such embodiments, a system includes a lookup table,for example, stored in memory, for determining which of a plurality ofback-propagation algorithms and acoustic sensor arrays to use fordetermining acoustic image data. FIG. 8 shows an exemplary lookup tablefor determining an appropriate algorithm and sensor array for use duringan acoustic imaging process.

In the illustrated example, the lookup table of FIG. 8 includes Ncolumns, each representing different arrays: Array 1, Array 2, . . . ,Array N. In various examples, each array comprises a unique set ofacoustic sensor elements arranged. Different arrays may comprise sensorelements arranged into a lattice (e.g., array 392 and array 396 in FIG.3C). Arrays within the lookup table can also include combinations ofsensor elements from one or more such lattices. In general, in someembodiments, each of arrays Array 1, Array 2, . . . , Array Ncorresponds to a unique combination of acoustic sensor elements. Some ofsuch combinations can include an entire set of sensor elements arrangedin a particular lattice, or can include a subset of sensor elementsarranged in a particular lattice. Any of a variety of combinations ofacoustic sensor elements are possible options for use as a sensor arrayin the lookup table.

The lookup table of FIG. 8 further includes M rows, each representing adifferent algorithm: Algorithm 1, Algorithm 2, . . . , Algorithm M. Insome examples, different algorithms may include different processes forperforming back-propagation analysis of received acoustic signals. Asdescribed elsewhere herein, in some examples, some different algorithmscan be similar to one another while having different coefficients and/orterms for modifying the back-propagation results.

The exemplary lookup table of FIG. 8 includes M×N entries. In someembodiments, the acoustic analysis system utilizing such a lookup tableis configured to analyze received distance information and classify thedistance information into one of M×N bins, wherein each bin correspondsto an entry in the lookup table of FIG. 8. In such examples, when theacoustic analysis system receives distance information, the system canfind the entry (i, j) in the lookup table corresponding to the bin inwhich the distance information lies, and determine the appropriatealgorithm and sensor array for use during an acoustic imaging process.For example, if the received distance information corresponds to the binassociated with entry (i, j), the acoustic analysis system canautomatically utilize or suggest using Algorithm i and Array j for anacoustic imaging process.

In various such examples, distance information bins can correspond touniformly sized distance ranges, for example, a first bin correspondingto distances of within one foot, a second bin corresponding to distancesbetween one and two feet, and so on. In other examples, bins need notcorrespond to uniformly sized distance spans. Additionally, in someembodiments, fewer than M×N bins can be used. For example, in someembodiments, there may be an algorithm (e.g., Algorithm x) that is notever used with a particular array (e.g., Array y). Thus, in such anexample, there would be no corresponding distance information bincorresponding to entry (x, y) in the M×N lookup table.

In some embodiments, statistical analysis on the populated distance binscan be used for identifying a most common distance or distance rangewithin the target scene. In some such embodiments, the distance binhaving the highest number of corresponding locations (e.g., a highestnumber of locations with acoustic signals) can be used as distanceinformation in the process of FIG. 7. That is, in some embodiments, autilized acoustic sensor array and/or processing scheme may beimplemented and/or recommended based on statistical analysis of thedistance distribution of various objects in the target scene. This canincrease the likelihood that sensor array and/or processing scheme usedfor acoustic imaging of a scene is appropriate for the largest number oflocations within the acoustic scene.

Additionally or alternatively, parameters other than distanceinformation can be used to select appropriate sensor arrays and/orprocessing schemes to use in generating acoustic image data. Asdescribed elsewhere herein, various sensor arrays can be configured tobe sensitive to certain frequencies and/or frequency bands. In someexamples, different back-propagation calculations similar can be usedaccording to different acoustic signal frequency content. Thus, in someexamples, one or more parameters can be used to determine a processingscheme and/or acoustic sensor array.

In some embodiments, the acoustic analysis system can be used toinitially analyze various parameters of received acoustic signalsprocessing/analysis. With reference back to FIG. 7, a method forgenerating acoustic image data can include a step of, after receivingacoustic signals (786), analyzing frequency content of the receivedsignals (790). In some such examples, if the acoustic sensor array(s)and/or processing schemes have been selected (e.g., via steps 782 and/or784, respectively), the method can include the steps of updating theselected array(s) and/or updating the selected processing scheme (792),for example, based on the analyzed frequency content.

After updating the sensor array(s) and/or processing scheme, the methodcan perform various actions using the updated parameters. For example,if the selected sensor array(s) is updated (792) based on analyzedfrequency content (790), then new acoustic signals can be received fromthe (newly) selected acoustic sensor array (786), which can then beback-propagated to determine acoustic image data (788). Alternatively,if the processing scheme is updated at 792, already-captured acousticsignals can be back-propagated according to the updated processingscheme to determine updated acoustic image data. If both the processingscheme and the sensor array(s) are updated, new acoustic signals can bereceived using the updated sensor array and can be back-propagatedaccording to the updated processing scheme.

In some embodiments, an acoustic analysis system can receive frequencyinformation (778) without analyzing frequency content of receivedacoustic signals (790). For instance, in some examples, an acousticanalysis system can receive information regarding a desired or expectedfrequency range for future acoustic analysis. In some such examples, thedesired or expected frequency information can be used to select one ormore sensor arrays and/or a processing scheme that best fits thefrequency information. In some such examples, the step(s) of selectingacoustic sensor array(s) (782) and/or selecting a processing scheme(784) can be based on received frequency information in addition oralternatively to received distance information.

In some examples, received acoustic signals (e.g., received via theacoustic sensor elements) can be analyzed, for example, via a processor(e.g., 210) of an acoustic analysis system. Such analysis can be used todetermine one or more properties of the acoustic signals, such asfrequency, intensity, periodicity, apparent proximity (e.g., a distanceestimated based on received acoustic signals), measured proximity, orany combinations thereof. In some examples, acoustic image data can befiltered, for instance, to only show acoustic image data representingacoustic signals having a particular frequency content, periodicity, orthe like. In some examples, any number of such filters can be appliedsimultaneously.

As described elsewhere herein, in some embodiments, a series of framesof acoustic image data can be captured over time, similar to acousticvideo data. Additionally or alternatively, even if acoustic image datais not repeatedly generated, in some examples, acoustic signals arerepeatedly sampled and analyzed. Thus, with or without repeated acousticimage data generation (e.g., video), parameters of acoustic data, suchas frequency, can be monitored over time.

FIG. 9A is an exemplary plot of frequency content of received image dataover time in an acoustic scene. As shown, the acoustic scene representedby the plot of FIG. 9A generally includes four sustained frequenciesover time, labeled as Frequency 1, Frequency 2, Frequency 3, andFrequency 4. Frequency data, such as the frequency content of a targetscene, can be determined via processing received acoustic signals, forexample, using a Fast Fourier Transform (FFT) or other known method offrequency analysis.

FIG. 9B shows an exemplary scene including a plurality of locationsemitting acoustic signals. In the illustrated image, acoustic image datais combined with visible light image data, and shows acoustic signalspresent at locations 910, 920, 930, and 940. In some embodiments,acoustic analysis system is configured to display acoustic image data ofany detected frequency range. For instance, in an exemplary embodiment,location 910 includes acoustic image data including Frequency 1,location 920 includes acoustic image data including Frequency 2,location 930 includes acoustic image data including Frequency 3, andlocation 940 includes acoustic image data including Frequency 4.

In some such examples, displaying acoustic image data representativefrequency ranges is a selectable mode of operation. Similarly, in someembodiments, acoustic analysis system is configures to display acousticimage data representative of frequencies only within a predeterminedfrequency band. In some such examples, displaying acoustic image datarepresenting a predetermined frequency range comprises selecting one ormore acoustic sensor arrays for receiving acoustic signals from which togenerate acoustic image data. Such arrays can be configured to receive aselective frequency range. Similarly, in some examples, one or morefilters can be employed to limit the frequency content used to generatethe acoustic image data. Additionally or alternatively, in someembodiments, acoustic image data comprising information representativeof a broad range of frequencies can be analyzed and shown on the displayonly if the acoustic image data satisfies a predetermined condition(e.g., falls within a predetermined frequency range).

FIG. 9C shows a plurality of combined acoustic and visible light imagedata at a plurality of predefined frequency ranges. A first imageincludes acoustic image data at a first location 910 that includesfrequency content of Frequency 1. A second image includes acoustic imagedata at a second location 920 that includes frequency content ofFrequency 2. A third image includes acoustic image data at a thirdlocation 930 that includes frequency content of Frequency 3. A fourthimage includes acoustic image data at a fourth location 940 thatincludes frequency content of Frequency 4.

In an exemplary embodiment, a user may select various frequency ranges,such as ranges including Frequency 1, Frequency 2, Frequency 3, orFrequency 4, for filtering acoustic image data representative offrequency content other than the selected frequency range. Thus, in suchexamples, any of the first, second, third, or fourth images may bedisplayed as a result of a desired frequency range being selected by auser.

Additionally or alternatively, in some examples, an acoustic analysissystem may cycle between a plurality of display images, each havingdifferent frequency content. For instance, with respect to FIG. 9C, inan exemplary embodiment, an acoustic analysis system may display, in asequence, the first, second, third, and fourth images, such as shown bythe arrows in FIG. 9C.

In some examples, display images can includes a text or other displayrepresentative of the frequency content being displayed in the image sothat a user may observe which locations in the image include acousticimage data representative of certain frequency content. For example,with respect to FIG. 9C, each image may show a textual representation ofthe frequency represented in the acoustic image data. With respect toFIG. 9B, an image showing a plurality of frequency ranges may includeindications of the frequency content at each location including acousticimage data. In some such examples, a user may select a location in theimage, for example, via a user interface, for which to view thefrequency content present at that location it the acoustic scene. Forexample, a user may select first location 910, and the acoustic analysissystem may present the frequency content of the first location (e.g.,Frequency 1). Thus, in various examples, a user can use the acousticanalysis system in order to analyze the frequency content of an acousticscene, such as by viewing where in the scene corresponds to certainfrequency content and/or by viewing what frequency content is present atvarious locations.

During exemplary acoustic imaging operation, filtering acoustic imagedata by frequency can help reduce image clutter, for example, frombackground or other unimportant sounds. In an exemplary acoustic imagingprocedure, a user may wish to eliminate background sounds, such as floornoise in an industrial setting. In some such instances, background noisecan include mostly low frequency noise. Thus, a user may choose to showacoustic image data representative of acoustic signals greater than apredetermined frequency (e.g., 10 kHz). In another example, a user maywish to analyze a particular object that generally emits acousticsignals within a certain range, such as corona discharge from atransmission line (e.g., as shown in FIGS. 5A-D5). In such an example, auser may select a particular frequency range (e.g., between 11 kHz and14 kHz for corona discharge) for acoustic imaging.

In some examples, an acoustic analysis system can be used to analyzeand/or present information associated with the intensity of receivedacoustic signals. For example, in some embodiments, back-propagatingreceived acoustic signals can include determining an acoustic intensityvalue at a plurality of locations in the acoustic scene. In someexamples, similar to frequency described above, acoustic image data isonly included in a display image if the intensity of the acousticsignals meets one or more predetermined requirements.

In various such embodiments, a display image can include acoustic imagedata representative of acoustic signals above a predetermined threshold(e.g., 15 dB), acoustic signals below a predetermined threshold (e.g.,100 dB), or acoustic signals within a predetermined intensity range(e.g., between 15 dB and 40 dB). In some embodiments, a threshold valuecan be based on a statistical analysis of the acoustic scene, such asabove or below a standard deviation from the mean acoustic intensity.

Similar to as described above with respect to frequency information, insome embodiments, restricting acoustic image data to represent acousticsignals satisfying one or more intensity requirements can includefiltering received acoustic signals so that only received signals thatsatisfy the predetermined conditions are used to generate acoustic imagedata. In other examples, acoustic image data is filtered to adjust whichacoustic image data is displayed.

Additionally or alternatively, in some embodiments, acoustic intensityat locations within an acoustic scene can be monitored over time (e.g.,in conjunction with a video acoustic image representation or viabackground analysis without necessarily updating a display image). Insome such examples, predetermined requirements for displaying acousticimage data can include an amount or rate of change in acoustic intensityat a location in an image.

FIGS. 10A and 10B are exemplary display images including combinedvisible light image data and acoustic image data. FIG. 10A shows adisplay image including acoustic image data shown at a plurality oflocations 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090. Insome examples, intensity values can be palettized, for example, whereinan acoustic intensity value is assigned a color based on a predeterminedpalettization scheme. In an exemplary embodiment, intensity values canbe categorized according to intensity ranges (e.g., 10 dB-20 dB, 20dB-30 dB, etc.). Each intensity range can be associated with aparticular color according to a palettization scheme. Acoustic imagedata can include a plurality of pixels, wherein each pixel is colorizedin the color associated with the intensity range into which theintensity represented by the pixel of acoustic image data falls. Inaddition or alternatively to being differentiated by color, differentintensities can be distinguished according to other properties, such astransparency (e.g., in an image overlay in which acoustic image data isoverlaid onto other image data) or the like.

Additional parameters may also be palettized, such as a rate of changeof acoustic intensity. Similar to intensity, varying rates of change inacoustic intensity can be palettized such that portions of the scenesexhibiting different rates and/or amounts of acoustic intensity changeare displayed in different colors.

In the illustrated example, the acoustic image data is palettizedaccording to an intensity palette, such that acoustic image datarepresentative of different acoustic signal intensities are shown in adifferent color and/or shade. For instance, acoustic image data atlocations 1010 and 1030 show a palletized representation of a firstintensity, locations 1040, 1060, and 1080 show a palletizedrepresentation of a second intensity, and locations 1020, 1050, 1070,and 1090 show a palletized representation of a third intensity. As shownin the exemplary representation in FIG. 10A, each location showing apalettized representation of acoustic image data shows circular patternhaving a color gradient extending outward from the center. This can bedue to the decay of acoustic intensity as the signals propagate from asource of the acoustic signals.

In the example of FIG. 10A, acoustic image data is combined with visiblelight image data to generate a display image, which may be presented toa user, for example, via a display. A user may view the display image ofFIG. 10A in order to view which locations in a visible scene areproducing acoustic signals, and the intensities of such signals. Thus, auser may quickly and easily observe which locations are producing soundsand compare the intensities of the sounds coming from various locationsin the scene.

Similar to as described with respect to frequencies elsewhere herein, insome embodiments, acoustic image data may be presented only if thecorresponding acoustic signals meet a predetermined intensity condition.FIG. 10B shows an exemplary display image similar to the display imageof FIG. 10A and including visible light image data and acoustic imagerepresenting acoustic signals above a predetermined threshold. As shown,of locations 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090 inFIG. 10A that include acoustic image data, only locations 1020, 1050,1070, and 1090 include acoustic image data representing acoustic signalsthat meet a predetermined condition.

In an exemplary scenario, FIG. 10A can include all acoustic image dataabove a noise floor threshold at each of locations 1010-990, while FIG.10B shows the same scene as FIG. 10A, but only showing acoustic imagedata having an intensity greater than 40 dB. This can help a useridentify which sources of sound in an environment (e.g., in the targetscene of FIGS. 10A and 10B) are contributing certain sounds (e.g., theloudest sounds in a scene).

In addition or alternatively to being compared directly to an intensitythreshold (e.g., 40 dB), as described elsewhere herein, in some suchexamples, predetermined requirements for displaying acoustic image datacan include an amount or rate of change in acoustic intensity at alocation in an image. In some such examples, acoustic image data may bepresented only if a rate of change or an amount of change in acousticintensity at a given location satisfies a predetermined condition (e.g.,is greater than a threshold, less than a threshold, within apredetermined range, etc.). In some embodiments, amount or rate ofchange of acoustic intensity can be palettized and displayed as or inconjunction with intensity acoustic image data. For instance, in anexemplary embodiment, when a rate of change is used as a threshold todetermine in which locations to include acoustic image data, theacoustic image data can include a palettized intensity rate of changemetric for display.

In some examples, a user may manually set an intensity requirement(e.g., minimum value, maximum value, range, rate of change, amount ofchange, etc.) for the acoustic image data to be displayed. As discussedelsewhere herein, including acoustic image data that only meets theintensity requirement can be achieved during acoustic image datageneration (e.g., via filtering received acoustic signals) and/or can beperformed by not displaying generated acoustic image data representingacoustic signals that do not meet the set requirement(s). In some suchexamples, filtering a display image according to intensity values can beperformed after the acoustic image data and visible light image datahave been captured and stored in memory. That is, data stored in memorycan be used to generate display images including any number of filteringparameters, such as only showing acoustic image data meeting predefinedintensity conditions and the like.

In some examples, setting a lower bound for intensity in an acousticimage (e.g., only displaying acoustic image data representative ofacoustic signals above a predetermined intensity) can eliminate theinclusion of undesired background or ambient sounds and/or soundreflections from the acoustic image data. In other instances, setting anupper bound for intensity in an acoustic image (e.g., only displayingacoustic image data representative of acoustic signals below apredetermined intensity) can eliminate the inclusion of expected loudsounds in acoustic image data in order to observe acoustic signalsordinarily masked by such loud sounds.

Several display functions are possible. For example, similar to thefrequency analysis/display discussed with respect to FIG. 9C, in someexamples, the acoustic analysis system can cycle through a plurality ofdisplay images, each showing acoustic image data satisfying differentintensity requirements. Similarly, in some examples, a user may scrollthrough a series of acoustic intensity ranges in order to view thelocations in the acoustic image data having acoustic intensity in thegiven range.

Another parameter that can be used to analyze acoustic data is aperiodicity value of an acoustic signal. FIGS. 11A and 11B showexemplary plots of frequency vs. time of acoustic data in an acousticscene. As shown in the plot of FIG. 11A, the acoustic data includessignals at a frequency X having a first periodicity, signals at afrequency Y having a second periodicity, and signals at a frequency Zhaving a third periodicity. In the illustrated example, acoustic signalshaving different frequencies may also include different periodicity inthe acoustic signals.

In some such examples, acoustic signals can be filtered based onperiodicity in addition or alternatively to frequency content. Forinstance, in some examples, multiple sources of acoustic signals in anacoustic scene may produce acoustic signals at a particular frequency.If a user wishes to isolate one such sound source for acoustic imaging,the user may choose to include or exclude acoustic image data from afinal display image based on the periodicity associated with theacoustic data.

FIG. 11B shows a plot of frequency vs. time of an acoustic signal. Asshown, the frequency increases over time approximately linearly.However, as shown, the signal includes an approximately constantperiodicity over time. Thus, such a signal may or may not appear in anacoustic image depending on selected display parameters. For instance,the signal may at some points in time satisfy a frequency criteria forbeing displayed, but at other points in time, be outside of a displayedfrequency range. However, a user could choose to include or exclude sucha signal from acoustic image data based on the periodicity of thesignals regardless of the frequency content.

In some examples, extracting acoustic signals of a particularperiodicity can be helpful in analyzing a particular portion of a targetscene (e.g., a particular piece of equipment or type of equipment thattypically operates at a certain periodicity). For example, if an objectof interest operates at a certain periodicity (e.g., once per second),excluding signals having periodicity distinct from this can improveacoustic analysis of the object of interest. For example, with referenceto FIG. 11B, if an object of interest operates at periodicity 4,isolating signals having periodicity 4 for analysis may yield improvedanalytics of the object of interest. For example, the object of interestmay emit sounds having periodicity 4, but increasing frequency, such asshown in FIG. 11B. This can imply that the properties of the object maybe changing (e.g., increased torque or load, etc.) and should beinspected.

In an exemplary acoustic imaging process, background noises (e.g., floornoise in an industrial setting, wind in an outdoor environment, etc.)are generally not periodic while certain objects of interest within ascene emit period acoustic signals (e.g., machinery operating at aregular interval). Thus, a user may choose to exclude non-periodicacoustic signals from an acoustic image in order to remove backgroundsignals and more clearly present acoustic data of interest. In otherexamples, a user may be looking to find the source of a constant tone,and so may choose to exclude period signals from acoustic image datathat may obscure viewing of a constant tone. In general, a user maychoose to include in acoustic image data acoustic signals that are abovea certain periodicity, below a certain periodicity, or within a desiredrange of periodicities. In various examples, periodicity can beidentified by either a length of time between periodic signals or afrequency of occurrence of periodic signals. Similar to frequency asshown in FIG. 11B, an analysis of intensity at a given periodicity(e.g., due to an object of interest operating at that periodicity) cansimilar be used to track how acoustic signals from the object changeover time. In general, in some embodiments, periodicity can be used toperform rate-of-change analysis for a variety of parameters, such asfrequency, intensity, and the like.

As describe elsewhere herein, in some examples, various portions of atarget scene can be associated with different distances from an acousticimaging sensor. For example, in some embodiments, distance informationcan include three-dimensional depth information regarding variousportions in a scene. Additionally or alternatively, a user may be ableto measure (e.g., with a laser distance tool) or manually input distancevalues associated with a plurality of locations in a scene. In someexamples, such different distance values for various portions of thescene can be used to adjust the back-propagation calculations at suchlocations to accommodate the specific distance value at that location.

Additionally or alternatively, if different portions of the scene areassociated with different distance values, then proximity from theacoustic sensor array (e.g., measured proximity and/or apparentproximity) can be another differentiable parameter between suchportions. For example, with respect to FIG. 10B, locations 1020, 1050,1070, and 1090 are each associated with a different distance value. Insome examples, similar to frequency or periodicity discussed elsewhereherein, a user can select a particular distance range from which toinclude acoustic image data on a display. For example, a user may selectto only display acoustic image data representative of acoustic signalscloser than a predetermined distance, further than a predetermineddistance, or within a predetermined range of distances.

In addition, in some embodiments, similar to as described with respectto frequencies in FIG. 9C, an acoustic analysis system can be configuredto cycle through a plurality of distance ranges, only showing acousticimage data representing acoustic signals emitted from a location in thetarget scene meeting a current distance range. Such cycling throughvarious displays can help a user visually distinguish informationbetween different acoustic signals. For example, in some cases, objectsmay appear to be close together from the line of sight from anassociated electromagnetic imaging tool (e.g., a visible light cameramodule), and thus acoustic image data combined with electromagneticimage data of such objects may be difficult to distinguish. However, ifthe objects are separated by a depth difference, cycling throughdifferent depth ranges of acoustic image data can be used to isolateeach source of acoustic data from the other.

In general, an acoustic analysis system can be configured to applyvarious settings in order to include and/or exclude acoustic image datarepresentative of acoustic signals that meet one or more predefinedparameters. In some examples, acoustic analysis system can be used toselect a plurality of conditions which must be met by acoustic signalsin order for acoustic image data representative of such signals isdisplayed, for example, in a display image.

For instance, with respect to FIGS. 10A and 10B, only acoustic signalsabove a threshold intensity in the scene of FIG. 10A are shown in FIG.10B. However, additional or alternative restriction is possible. Forexample, in some embodiments, a user may additionally filter theacoustic image data so that acoustic image data is only shown foracoustic signals having frequency content within a predeterminedfrequency range and/or having a predetermined periodicity. In anexemplary embodiment, restricting to predetermined frequencies and/orperiodicities of interest, acoustic image data may be eliminated fromadditional locations, such as 1020 and 1090.

In general, a user can apply any number of acoustic data requirementsfor including or excluding acoustic image data from a display image,including parameters such as intensity, frequency, periodicity, apparentproximity, measured proximity, sound pressure, particle velocity,particle displacement, sound power, sound energy, sound energy density,sound exposure, pitch, amplitude, brilliance, harmonics, rates of changeof any such parameters, or the like. Additionally, in some embodiments,user may combine requirements using any appropriate logicalcombinations, such as AND, OR, XOR, etc. For instance, a user may wishto display only acoustic signals having (intensity above a predeterminedthreshold) AND (frequency within a predetermined range).

Additionally or alternatively, the acoustic analysis system can beconfigured to cycle through one or more parameter ranges to illustratedifferent portions of the target scene, such as shown with respect tocycling through a plurality of frequencies in FIG. 9C. In general, oneor more parameters can be cycled through in such a manner. For instance,a parameter (e.g., intensity) can be separated into a plurality ofranges (e.g., 10 dB-20 dB and 20 dB-30 dB), and the acoustic analysissystem can cycle through such ranges, displaying all acoustic image datafalling within a first range, then all acoustic image data fallingwithin a second range, and so on.

Similarly, in some embodiments, an acoustic analysis system can beconfigured to combine parameter requirements by cycling through nestedranges. For instance, in an exemplary embodiment, acoustic image datathat satisfies a first intensity range AND a first frequency range canbe displayed. The displayed frequency range can be cycled through whilelimiting the displayed acoustic image data to acoustic signalssatisfying the first intensity range. After cycling through thefrequency ranges, the intensity range can be updated to a secondintensity range, such that the displayed acoustic image data satisfiesthe second intensity range and the first frequency range. Similar to theprocess incorporating the first intensity range, the frequency rangescan be similarly cycled through while maintaining the second intensityrange. This process can be continued until all combinations of frequencyranges and intensity ranges have been satisfied. Similar such processescan be performed for any of a plurality of parameters.

Additionally or alternatively, in some embodiments, an acoustic analysissystem can be configured to identify and distinguish a plurality ofsounds in the acoustic scene. For instance, with respect to FIG. 9B, theacoustic analysis system can be configured to identify four discretesounds at locations 910, 920, 930, and 940. The system can be configuredto cycle through a plurality of displays, each showing acoustic imagedata at a single discrete location, similar to as shown in FIG. 9C,though not necessarily dependent on any parameter values. Similarly,such cycling between acoustic image data at individual locations can beperformed after one or more parameter requirements limit the acousticimage data that is displayed.

For example, with respect to FIGS. 10A and 10B, before an intensitythreshold is applied, an acoustic analysis system may cycle through aplurality of acoustic image scenes (e.g., as display images includingthe acoustic image scenes with visible light image data), wherein eachscene includes acoustic image data at a single location. In someembodiments, according to the illustrated example of FIG. 10A, a cycleof 10 separate images, each image including image data at a differentone of locations 1010, 1020, 1030, 1040, 1050, 1060. 1070, 1080, and1090. However, according to some embodiments, after the intensity filteris applied so that only locations having intensity greater than athreshold are displayed (e.g., as in FIG. 10B), the acoustic analysissystem may update the cycling process to only cycle through imagescorresponding to locations that meet the filtering threshold. That is,with respect to FIG. 10B, the cycling process may update to only cyclebetween four images, each showing discrete acoustic image data atlocations 1020, 1050, 1070, and 1090, respectively.

Thus, in various embodiments, each of the locations in a target scenethat includes acoustic image data, either before or after applying oneor more filters to restrict which acoustic image data is shown, is shownin one of a plurality of cycled-through display images. Such cyclicaldisplay of individual acoustic source locations can assist a userviewing the images in identifying the source of particular sound. Insome embodiments, each image in the cycle includes only a single sourceof acoustic data, and in some such embodiments, further includes one ormore parameters of the acoustic data, such as frequency content,intensity, periodicity, apparent proximity, or the like.

In addition or alternatively to cycling between images showing acousticimage data satisfying certain conditions, in some examples, locations ofacoustic signal sources can be detected in acoustic image data anddisplayed in acoustic image data in isolation from other acousticsignals. For example, with respect to FIG. 10A, in some embodiments,acoustic image data representative of acoustic signals emanating fromeach of locations 1010-990 can be identified and cycled through. Forinstance, in an exemplary operating process, display images includingacoustic image data at one of locations 1010-990 can be cycled through,either automatically or at the direction of a user, for individualanalysis of each source of acoustic signals. In various embodiments, theorder in which different locations of acoustic image data are displayedwhile cycling can be dependent on a variety of parameters, such as bylocation, proximity, intensity, frequency content, or the like.

Additionally or alternatively, in some examples, acoustic image datafrom individual locations can by cycled through after applying one ormore filters to isolate only acoustic image data meeting one or morepredetermined conditions. For example, with respect to FIG. 10B,locations 1020, 1050, 1070, and 1090 are shown as including acousticimage data representing acoustic signals meeting a predeterminedintensity requirement. In some embodiments, such a display requirementcan be applied to the individual cycling through of source locations ofacoustic signals. For example, with further reference to FIG. 10B,display images including image data from only one of locations 1020,1050, 1070, and 1090 satisfying an acoustic intensity condition can bycycled through for individual analysis at each location.

In an exemplary process with reference to FIGS. 10A and 10B, acousticimage data collected from a scene can be generally shown in FIG. 10A atlocations 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090. Suchlocations can include acoustic image data representative of acousticsignals having a variety of acoustic parameters, such as a variety ofintensities, frequency content, periodicity, and the like.

As described elsewhere herein, a user may wish to isolate acousticsignals having one or more particular acoustic parameters, such asacoustic signals having a minimum acoustic intensity. Acoustic imagedata representing acoustic signals not meeting such conditions can beexcluded from the image, for example, leaving acoustic image data atlocations 1020, 1050, 1070, and 1090 as shown in FIG. 10B. However, auser may wish to further identify the source of a particular soundmeeting the display condition (e.g., having intensity above athreshold). Thus, the user may choose to display the acoustic image dataassociated with locations 1020, 1050, 1070, and 1090 one-by-one in orderto view the source location of and analyze each sound individually. Invarious embodiments, the user may choose to cycle manually through suchlocations, or a processor may automatically update the display image tosequentially display acoustic image data of individual locations. Thismay help a user further eliminate and disregard acoustic signals not ofinterest, but that happen to meet one or more filtering parametersapplied to the image.

While described with respect to intensity and FIGS. 10A and 10B, ingeneral, display images including acoustic image data from a singlelocation selected from a plurality of locations can be cycled throughone-by-one for individual analysis. The plurality of locations for whichrepresentative acoustic image data is included can be the entire set oflocations corresponding to sources of acoustic signals in an acousticscene, or can be a subset of such locations, for example, including onlylocations having acoustic signals satisfying one or more conditions.Such conditions can depend on any one or more acoustic parameters, suchas intensity, frequency content, periodicity, proximity, or the like,and can be met based on various parameters being below a predeterminedvalue, above a predetermined value, or within a predetermined range ofvalues.

In various examples, modifying the display image to selectively includeacoustic image data in a display image can be done in a variety of ways.In some embodiments, display images (e.g., including electromagneticimage data and acoustic image data) can be real-time images, in whichelectromagnetic image data and acoustic image data is continuallyupdated to reflect changes in the scene. In some examples, when certainconditions are used to determine whether or not acoustic image data isincluded in the display image, received acoustic signals are analyzed todetermine whether or not to include acoustic image data at variouslocations in the updated real-time image. That is, as new display imageis generated based on newly received acoustic signals andelectromagnetic radiation, the construction of a display image candepend on analysis of the acoustic signals to determine which acousticsignals meet any specific conditions placed on the display image (e.g.,intensity thresholds, etc.). The display image can then be generatedincluding acoustic image data only where appropriate according to suchconditions.

In other examples, display images can be generated from data stored inmemory, such as previously captured acoustic data and electromagneticimage data. In some such examples, the previously-acquired acoustic datais analyzed with respect to various conditions to be placed on theacoustic image data, and is combined with electromagnetic image data inlocations in which the previously-captured acoustic data meets suchconditions. In such embodiments, a single scene can be viewed in manyways, for example, by analyzing different acoustic parameters. Thedisplay image representative of the previously-captured acoustic imagedata can be updated based on any updated conditions placed on thedisplay image for whether or not to include acoustic image data invarious locations in the display image.

In some embodiments, one or more acoustic parameters used to selectivelyinclude acoustic image data in a display image may be used to modify thedisplay image and/or image capturing techniques. For example, inreal-time imaging examples, various conditions for determining whetheror not to include acoustic image data in a display can includedistance-to-target (e.g., apparent distance or measured distance) and/orfrequency content. As described elsewhere herein, some such parameterscan be used in selecting an acoustic sensor array and/or processingscheme for generating acoustic image data. Thus, in some such examples,when acoustic image data is only represented based on such parametersmeeting one or more predetermined conditions, an acoustic sensor arrayand/or a processing scheme for generating acoustic image data can beselected based on such conditions.

For example, in an exemplary embodiment, if acoustic image data is onlyto be included in a real-time image in locations at which correspondingacoustic signals include frequency content within a first frequencyrange, one or more acoustic sensor arrays can be selected for acquiringacoustic signals that are best suited for the first frequency range.Similarly, if acoustic image data is only to be included in a real-timeimage at locations in which a source of acoustic signals is within afirst distance range, one or more acoustic sensor arrays can be selectedfor acquiring acoustic signals that are best suited for acoustic imagingin the first distance range. Additionally or alternatively, asdescribed, for example, with respect to FIG. 6, processing schemes forgenerating acoustic image data can be selected based on desiredfrequency or distance conditions. Such selected acoustic imaging sensorarray(s) sand processing schemes can be subsequently used to receiveacoustic signals and generate acoustic image data for the updatedreal-time display image in order to optimize the acoustic image datathat is included.

Similarly, in some embodiments in which a display image is generatedfrom historical data previously stored in memory, various conditionsdetermining in which locations to include acoustic image data in thedisplay image can be used to update the acoustic image datarepresentative of the acoustic scene. For instance, in some embodiments,data stored in memory comprises raw acoustic data received by theacoustic sensor array(s) from the time the acoustic signals werereceived. Based on the conditions for determining whether or notacoustic image data is included at various locations in the displayimage (e.g., desired distance and/or frequency ranges), a processingscheme (e.g., a back-propagation algorithm) can be selected for use withthe raw data stored in memory for generating acoustic image dataoptimized to the desired parameters to be displayed.

It will be appreciated that, while generally described and shown usingvisible light image data and acoustic image data, the processesdescribed with respect to FIGS. 9A-C, 10A and 10B can be used includingany of a variety of electromagnetic image data. For example, in variousembodiments, similar processes could be performed with infrared imagedata or ultraviolet image data instead of visible light image data.Additionally or alternatively, combinations of electromagnetic spectrumscan be used in such processes, such as blended infrared image data andvisible light image data. In general, in various examples, acousticimage data can be selectively shown (e.g., included when correspondingacoustic signals meet one or more predetermined parameters) incombination any combination of electromagnetic image data.

In some embodiments, an acoustic analysis system is configured to storeone or more acoustic signals and/or acoustic image data in a database,for example, in local memory and/or accessible from an external orremote device. Such acoustic signals can include acoustic image datarepresentative of an acoustic scene during normal operation and/or otherparameters associated with an acoustic scene, such as frequency data,intensity data, periodicity data, and the like. In various examples,database scenes can include acoustic image data and/or other acousticparameters (e.g., intensity, frequency, periodicity, etc.)representative of a broad scene (e.g., a factory) and/or a more specificscene (e.g., a particular object).

In some embodiments, a database scene can be generic to a particulartype of equipment, such as a particular model of equipment. Additionallyor alternatively, database scenes can be specific to individual objects,even if different such objects are different instances of the sameobject (e.g., two separate machines that are the same model). Similarly,database scenes can be more specific, for example, including aparticular operating state of an object. For instance, if a particularobject has multiple modes of operation, a database can include multiplescenes of such an object, one for each of the modes of operation.

In various embodiments, database scenes can be a single acoustic imageand/or associated acoustic parameters. In other examples, databasescenes can include composite data formed from a plurality of previouslycaptured acoustic images and/or associated parameters. In general,database scenes (e.g., acoustic images and/or parameters) can include anacoustic representation of the scene during normal operation. In someexamples, the database can include other elements associated with thescene, such as a corresponding visible light image, infrared image,ultraviolet image, or combinations thereof, for example. In someembodiments, database generation and/or comparisons can be performedsimilar to the database generation and comparisons of infrared imagedata described in U.S. patent application Ser. No. 15/190,792, filedJun. 23, 2016, and entitled “THERMAL ANOMALY DETECTION,” which isassigned to the assignee of the instant application and is herebyincorporated by reference in its entirety. In some embodiments, adatabase can be generated by capturing acoustic image data and/or one ormore associated acoustic parameters (e.g., frequency, intensity,periodicity, etc.) of a scene while objects within the scene areoperating correctly. In some such examples, a user may tag the captureddatabase image to associate the image with one or more objects,locations, scenes, or the like, so that the captured acoustic imageand/or associated parameter(s) can be identified in the future fordatabase analysis and comparisons.

Newly generated acoustic image data can be compared to acoustic imagedata stored in the database to determine whether or not the acousticprofile of the acoustic scene is within typical operating standards.Additionally or alternatively, acoustic parameters, such as intensity,frequency, periodicity, and the like, from a live acoustic scene and/ora newly-generated acoustic image can be compared to similar parametersin the database.

Comparing current acoustic image data to historical acoustic image data(e.g., a previously-captured image, a composite image generated from aplurality of previously-captured images, a factory-provided expectedimage, etc.) stored in a database can be done a plurality of ways. FIGS.12A-12C show multiple exemplary ways for comparing acoustic image datato historical acoustic image data stored in a database. FIG. 12A showsan acoustic imaging tool 1200 including an acoustic sensor array 1202having an acoustic field of view 1212 and an electromagnetic imagingtool 1204 having an electromagnetic field of view 1214. As shown, theelectromagnetic field of view 1214 and the acoustic field of view 1212include a target scene 1220 including an object of interest 1222. Insome embodiments, the acoustic imaging tool 1200 is permanently fixed ina location such that the object of interest 1222 is in theelectromagnetic field of view 1214 and the acoustic field of view 1212.In some embodiments, the acoustic imaging tool 1200 can be powered viainductive or parasitic power, can be wired into AC main power in abuilding, or the to be configured to continually monitor object 1222.

Fixed acoustic imaging tool 1200 can be configured to periodicallycapture acoustic and/or electromagnetic image data of object 1222.Because the acoustic imaging tool 1200 is generally fixed in place,images captured at different times will be from approximately the samevantage point. In some examples, acoustic image data captured viaacoustic imaging tool 1200 can be compared to a database of acousticimage data representative of approximately the same scene, for example,to detect anomalies or abnormalities in the acoustic scene. This can beperformed, for example, as described in U.S. patent application Ser. No.15/190,792, which is incorporated by reference.

FIG. 12B shows an exemplary display, for example, on a handheld acousticimaging tool. The display 1230 includes two sections, 1232 and 1234. Inthe illustrated example, section 1234 shows a database image 1244 of anobject of interest, while section 1232 comprises a live display 1242 ofreal-time acoustic image data of an object. In such a side-by-side view,a user may compare the live image 1242 to the database image 1244 inorder to view any differences between a typical acoustic signals (e.g.,as shown in database image 1244) and the current real-time image 1242.Similarly, the user can compare if the live image 1242 approximatelymatches the database image 1244. If so, the user may capture the liveacoustic image for further analysis and/or comparison to the databaseimage 1244.

FIG. 12C shows another exemplary display, for example, on a handheldacoustic imaging tool. The display 1250 of FIG. 12C shows a databaseimage 1254 and a live image 1256 on the same display 1252. In theexample of FIG. 12C, a user can similarly compare the acoustic imagedata in the live image 1256 to the acoustic image data in the databaseimage 1254 in order to view differences. Additionally, the user mayadjust alignment of the acoustic imaging tool in order to align theobject in the live image 1256 with the object in the database image 1254for further analysis and comparison.

As a result of the processes in FIGS. 12A-12C, live and/or recentlycaptured acoustic images can be compared to previous acoustic imagedata, such as from a database. In some examples, such processes can beused to register the live and/or recently captured acoustic image withthe database image for automated comparison. Other processes that can beused to “recapture” acoustic image data from a similar vantage point asthe database image are described in U.S. patent application Ser. No.13/331,633, filed Dec. 20, 2011, and entitled, “THERMAL IMAGING CAMERAFOR INFRARED REPHOTOGRAPHY,” U.S. patent application Ser. No.13/331,644, filed Dec. 20, 2011, and entitled, “THERMAL IMAGING CAMERAFOR INFRARED REPHOTOGRAPHY,” and U.S. patent application Ser. No.13/336,607, filed Dec. 23, 2011, and entitled, “THERMAL IMAGING CAMERAFOR INFRARED REPHOTOGRAPHY,” each of which is assigned to the assigneeof the instant application and is incorporated by reference in itsentirety.

Comparing real-time acoustic image data and/or acoustic signatures to acorresponding acoustic image and/or acoustic signature of a comparablescene/object can be used to provide fast and simplified analysis of thestate of operation of the scene/object. For example, a comparison mayindicate that certain locations within the acoustic scene are emittingacoustic signals that have a different intensity or frequency spectrumthan during typical operation, which can be indicative of a problem.Similarly, locations in the scene may be emitting acoustic signals thatare typically silent. Additionally or alternatively, comparison ofoverall acoustic signatures of a live scene and a historic scene from adatabase can generally indicate changes in acoustic parameters in thescene, such as frequency content, acoustic intensity, and the like.

In some examples, an acoustic analysis system is configured to comparethe recent/real-time acoustic scene with a database. In someembodiments, the acoustic analysis system is configured to characterizethe differences between the recent/real-time scene and the databasescene and diagnose one or more possible problems in the current scenebased on the comparison. For instance, in some embodiments, a user maypre-select an object of interest or a target scene for comparison to anacoustic database. The acoustic analysis system can, based on theselected object/scene, compare the database image and/or otherparameters to the recent/current image and/or other parameters toanalyze the scene. Based on the selected object/scene from the database,the acoustic analysis system may be capable of identifying one or moredifferences between the database image/parameters and the recent/currentimage/parameters and associate the identified difference(s) with one ormore likely causes of the differences.

In some examples, the acoustic analysis system can be pre-programmedwith a plurality of diagnostic information, for example, associatingvarious differences between database images/parameters andrecent/current images/parameters with likely causes and/or solutions tocauses. Additionally or alternatively, a user may load such diagnosticinformation, for example, from a repository of diagnostic data. Suchdata may be provided, for example, by a manufacturer of the acousticanalysis system, the manufacturer of an object of interest, or the like.In still further examples, an acoustic analysis system can additionallyor alternatively learn diagnostic information, for example, via one ormore machine learning processes. In some such examples, a user maydiagnose one or more issues in a target scene after observing acousticdeviations of the scene from typical, and may input data representativeof the one or more issues and/or one or more solutions into the acousticanalysis system. The system can be configured to, over time and viamultiple data entries, learn to associate different discrepanciesbetween recent/current images and/or parameters and those stored in adatabase with certain problems and/or solutions. Upon diagnosing anissue and/or determining a proposed solution, the acoustic analysissystem can be configured to output a suspected problem and/or proposedsolution to a user, for example, via a display. Such a display can be ona handheld acoustic inspection tool or a remote device (e.g., a user'ssmartphone, tablet, computer, etc.). Additionally or alternatively, sucha display indicating a potential problem and/or solution can becommunicated to a remote site, such as an off-site operator/systemmonitor, for example, via a network.

In some example diagnostic characterizations, an acoustic analysissystem may observe a particular periodic squeaking sound indicatingadditional lubrication is needed on an operating machine. Similarly, aconstant, high-pitched signal could indicate a gas or air leak in atarget scene. Other issues may similarly have recognizable acousticsignatures, such as a broken bearing within an object under analysis,such that viewing the acoustic signature via an acoustic imaging system(e.g., a handheld acoustic imaging tool) can help diagnose anyabnormalities in a system or object.

An acoustic analysis system capable of comparing received acousticsignals to a baseline (e.g., acoustic image data and/or parameters froma database) and performing diagnostic information and/or suggesting acorrective action can eliminate the need for an experienced expert toanalyze acoustic data of a scene. Rather, an acoustic inspection andanalysis can be performed by a system operator with limited or noexperience in analyzing acoustic data.

FIG. 13 is a process-flow diagram showing exemplary operation ofcomparing received acoustic image data to a database for objectdiagnostics. The method includes receiving a selection of a target ofinterest (1380) and retrieving a baseline acoustic image and/or acousticparameters of the target of interest from a database (1382). Forexample, a user may wish to perform acoustic analysis of a particularobject of interest, and may select such an object from a predefined listof objects having available baseline acoustic image and/or parametersavailable in the database.

The method further includes the step of capturing acoustic image dataand associated parameters representative of the target of interest(1384), for example, using a handheld acoustic imaging tool. Aftercapturing the acoustic image data and associated parameters (1384), themethod includes comparing the captured acoustic image data and/orassociated parameters to retrieved baseline image and/or parameters(1386).

The method of FIG. 13 further includes, if the captured acoustic imagedata and/or parameters deviate sufficiently from the baseline (1388),diagnosing operation issues of the target of interest based on thecomparison (1390). The method can further include the step of displayingan indication of possible issues and/or corrective actions to a user(1392). In some embodiments, the a comparison display, for example, adifference image showing the difference between the current acousticimage data and baseline acoustic image data can be additionally oralternatively displayed to a user.

In some such examples determining if there is deviation from thebaseline (1388) comprises comparing one or more acoustic parameters ofthe captured data to like parameters in the baseline data anddetermining if the difference between the captured and baselineparameters exceeds a predetermined threshold. In various examples,different parameters may include different thresholds, and suchthresholds can be absolute thresholds, statistical thresholds, or thelike. In some embodiments, comparisons can be done on alocation-by-location basis, and may be performed for a subset oflocations within a scene.

For example, with respect to FIG. 9B, it is possible that only locationsincluding acoustic image data and appearing on the object (e.g.,locations 910 and 940) are analyzed with respect to operation of theobject. In such an example, different acoustic parameters at each oflocations to be compared (e.g., 910 and 940) are compared individuallybetween captured and database images. For example, comparing thecaptured data and/or associated parameters to those from the databasecan include, with reference to FIG. 9B, comparing the frequency,intensity, and periodicity of location 910 in the captured image to thefrequency, intensity, and periodicity, respectively, of location 910 inthe database image. Similar comparisons can be performed at location 940between the captured image and the database image. As described, eachcomparison can include a different metric for determining if there issufficient deviation from the baseline (1388).

Diagnosing operation issues (1390) and displaying an indication ofpossible issues and/or corrective actions (1392) can be performed basedon the combination of comparisons between captured and baseline imagedata and/or parameters. In some examples, such diagnostics can include amulti-dimensional analysis, such as combining comparisons of multipleparameters at a given location. For instance, in an exemplaryembodiment, a certain condition might be indicated by both a deviationin frequency from the baseline that is greater than a first thresholdand a deviation in intensity from the baseline that is greater than asecond threshold.

In some examples, even after displaying an indication of possible issuesand/or corrective actions (1392), the process can include capturing newacoustic image data and associated parameters (1384) and repeating thecomparison and diagnostic processes. Thus, a user may observe whether ornot any taken corrective actions are effectively changing the acousticsignature of the object in order to rectify an identified issue and/orbring the acoustic signature of the object into conformity with thebaseline.

In some embodiments, if, after comparing the captured data to thebaseline data (1386), there is not a sufficient deviation from thebaseline (1388), the process may end (1394) with the conclusion that,based on the current acoustic signature of the object, the object isoperating normally. Additionally or alternatively, new acoustic imagedata and associated parameters of the target of interest can be captured(1384) and the comparison and diagnostic process can be repeated. Insome examples, continued repeated analysis can be performed using afixed acoustic analysis system, for example, including the acousticimaging tool 1200 in FIG. 12A.

Comparisons of acoustic data (e.g., image data and/or other acousticparameters) can help a user more easily identify if an object isfunction correctly, and if not, to diagnose issues with the object. Insome examples, comparing to a baseline can help a user disregard“normal” sounds in a scene, such as expected operating sounds orfloor/background sounds that may be irrelevant to an operating issue ofthe object.

During operation, observation of acoustic image data and/or associatedacoustic parameters or observing the results of a comparison betweencurrent and database acoustic scenes may indicate locations of interestto a user for further inspection. For example, a comparison acousticimage showing deviations from the database image may indicate one ormore locations in scene that are operating abnormally. Similarly,viewing an acoustic image having an acoustic signature at one or morelocations that are unexpected may indicate a location of interest to auser. For example, with reference to FIG. 10B, a user observing FIG. 10Bon a display of an acoustic imaging system may realize that a particularlocation (e.g., 1020) is unexpected emitting acoustic signals, orsimilarly, a comparison to a baseline image indicates unexpectedparameters of the acoustic signals at that location (e.g., unexpectedfrequency, intensity, or the like).

In some such examples, the user may move closer to such a location inorder to more closely inspect the location for abnormalities. Uponmoving closer to the object, the distance-to-target value may be updatedto reflect the new distance between an acoustic array and the targetlocation. The acoustic sensor array and/or the back-propagationalgorithm may be updated based on the updated distance-to-target.Additionally or alternatively, updated acoustic analysis from a closerlocation may yield different analysis of the acoustic signals from thetarget. For instance, high frequency acoustic signals (e.g., ultrasoundsignals) tend to attenuate over a relatively short distance from thesource of the acoustic signals. Thus, as a user moves closer to thetarget for further inspection, additional signals (e.g., high frequencysignals) may be visible to the acoustic sensor array. Such apparentchanges in the observable scene may also result in adjusting theacoustic sensor array and/or the back-propagation algorithm used foracoustic imaging.

Accordingly, the sensor array and/or back-propagation algorithm used foracoustic imaging can be updated one or more times as the user movescloser to an object or region of interest. Each update may provideadditional details regarding the object or region of interest that maynot have been observable from a further distance away using a differentsensor array and/or back-propagation algorithm. Moving closer to anobject or region of interest, for example, based on initial observationsof a broader scene, can also increase the acoustic intensity of theacoustic signals of interest relative to background sounds in theenvironment.

In some embodiments, an acoustic analysis system (e.g., a handheldacoustic imaging tool) can prompt a user to move more closely to anobject or regions of interest within a scene. For example, uponcomparing a current acoustic image to a baseline database image, theacoustic analysis system may identify one or more locations in the scenethat deviate from baseline. The acoustic analysis system may highlightsuch one or more locations to a user, for example, via a display, andsuggest the user move closer to the identified location(s) for furtheranalysis. In some examples, the acoustic analysis system can classifythe identified location, such as a sub-component of an object or aparticular object within an environment, as having its own baselineprofile stored in a database. The system may be configured to suggestand/or implement such a profile of the classified location to facilitatefurther analysis of the identified location when the user moves closerfor additional inspection.

Systems and processes described herein can be used to improve the speed,efficiency, accuracy, and thoroughness of acoustic inspections. Variousautomated actions and/or suggestions (e.g., of a sensor array, aback-propagation algorithm, etc.) can increase the ease of inspection tothe point that an inexperienced user may perform a thorough acousticinspection of an acoustic scene. Moreover, such processes can be used toanalyze a broad scope of scenes, such as entire systems, individualobjects, and sub-components of individual objects. Predefined and/oruser-generated profiles of baseline acoustic data of acoustic scenes canassist even inexperienced users in identifying abnormalities in capturedacoustic data.

Registration of acoustic image data with other data streams, such asvisible light, infrared, and/or ultraviolet image data, can provideadditional context and detail to what objects are emitting acousticsignals represented in acoustic image data. Combining acoustic sensorarrays and a distance measuring tool (e.g., a laser distance finder) canassist a user in quickly and easily determining a properdistance-to-target value for use during acoustic imaging processes. Invarious examples, an acoustic sensor array, distance measuring tool,processor, memory, and one or more additional imaging tools (e.g.,visible light camera module, infrared camera module, etc.) can besupported by a single housing in a handheld acoustic imaging tool thatcan provide efficient acoustic analysis of a plurality of scenes. Such ahandheld acoustic imaging tool can be moved from scene to scene forrapid analysis of multiple objects of interest. Similarly, using ahandheld tool, a user can move closer to a location of interest within ascene for further inspection or analysis.

Various systems and methods for performing acoustic imaging andgenerating and displaying acoustic image data are described herein.Exemplary systems can include an acoustic sensor array that includes aplurality of acoustic sensor elements configured to receive acousticsignals from an acoustic scene and output acoustic data based on thereceived acoustic signals.

Systems can include an electromagnetic imaging tool configured toreceive electromagnetic radiation from a target scene and outputelectromagnetic image data representative of the receivedelectromagnetic radiation. Such imaging tools can include an infraredimaging tool, a visible light imaging tool, an ultraviolet imaging tool,or the like, or combinations thereof.

Systems can include a processor in communication with the acousticsensor array and the electromagnetic imaging tool. The processor can beconfigured to receive electromagnetic image data from theelectromagnetic imaging tool and acoustic data from the acoustic sensorarray. The processor can be configured to generate acoustic image dataof a scene based on the received acoustic data and received distanceinformation representative of a distance to target, for example, via aback-propagation calculation. The acoustic image data can include avisual representation of the acoustic data such as by a palette or colorscheme such as described elsewhere herein.

The processor can be configured to combine the generated acoustic imagedata and the received electromagnetic image data to generate a displayimage comprising both acoustic image data and electromagnetic imagedata, and communicate the display image to a display. Combining theacoustic image data and the electromagnetic image data can includecorrecting a parallax error between the acoustic image data and theelectromagnetic image data, for example, based on the received distanceinformation.

In some examples, distance information can be received from a distancemeasuring tool in communication with the processor. Distance measuringtools can include, for example, an optical distance measuring device,such as a laser distance measuring device, and/or an acoustic distancemeasurement device. Additionally or alternatively, a user can enterdistance information manually, for example, via a user interface.

Systems can include a laser pointer to help identify locations of pointsof interest such as sounds or sound profiles based upon selectedparameters such as frequency, decibel level, periodicity, distance, orthe like, or combinations thereof. Such a laser pointer can be used topinpoint and align the field of view of the scene with the appropriatesound visualization as displayed on the display. This may be useful inenvironments where an object under inspection is at a distance relativeto the acoustic imaging device or if it is not clear where thevisualization of the sound on the display is relative to the actualscene.

In some examples, the laser pointer can be visualized on the display.Such visualization can include the generating a laser pointer spot(e.g., via the processor) on the display representative the laserpointer in the actual scene. In some examples, the position of the laserpointer can be enhanced on the display, for instance, with an iconrepresentative of the laser pointer in the scene or another aligneddisplay marker to better determine the location on the display relativeto the actual scene.

As described elsewhere herein, a thermal imaging system can beconfigured to create a false-color (e.g., palettized), symbolic, orother non-numerical visual representation of acoustic data generated byone or more acoustic sensors, such as by creating acoustic image data.Additionally or alternatively, a system can provide a user with audiofeedback, such as via speakers, headphones, a wired orremotely-communicating headset, or the like. The transmission of suchaudio or heterodyne audio can be synchronized to the visualrepresentation of the detected and displayed sounds.

In various examples, acoustic data can be visualized in a variety ofways, for example, to facilitate understanding of such data and preventa viewer from making incorrect assumptions about the nature of thesounds being visualized. In some examples, different types ofvisualization can provide an intuitive understanding of the visualizedsounds.

In some embodiments, a generated display includes a non-numerical visualrepresentation with contextual numerical and/or alphanumerical data inorder to provide a thorough presentation of information regarding soundsbeing visualized, which can assist a user in determining and/orimplementing one or more appropriate courses of action.

Various display features, including various non-numeric graphicalrepresentations (e.g., symbols, palettization, etc.) and alphanumericinformation can be combined. In some embodiments, the display featurespresent in a given representation of a scene can be customized by auser, for instance, from a plurality of selectable settings.Additionally or alternatively, preset combinations of display featurescan be selectable by a user to automatically include a desiredcombination of information in a display image. In various embodiments,aspects of a display image are adjustable by a user, for example, via avirtual interface (e.g., provided via a touchscreen) and/or via physicalcontrols.

FIG. 14 shows a visualization of acoustic data using a gradientpalettization scheme. As shown, an acoustic parameter (e.g., intensity)is shown via a gradient palettization scheme. In an exemplary gradientpalettization scheme, a parameter value will have a unique colorassociated therewith according to the palettization scheme. A change inthe parameter value at a given pixel will generally result in a changein the color associated with that pixel to represent the new parametervalue. As shown in the example of FIG. 14, the acoustic parameter valueappears to change radially from a center position in acoustic signal atpositions 232, 234, 236. Other examples of gradient palettization aredescribed U.S. patent application Ser. No. 15/802,153, filed Nov. 2,2017, and assigned to the assignee of the instant application.

FIG. 15 shows a visualization of acoustic data using a plurality ofshaded concentric circles. In some such examples, as opposed to agradient palettization scheme having a color associated with a parametervalue, each solid color the concentric circles shown in FIG. 15 canrepresent pixels having acoustic parameter values within a range ofvalues associated with that color. In the illustrated examples, anacoustic parameter (e.g., intensity) associated with acoustic signals atlocations 332, 334, 336 changes radially from the center of the acousticsignal. In an exemplary palettization scheme, pixels shown in redrepresent an acoustic parameter value within a first range of parametervalues, pixels shown in yellow represent an acoustic parameter valuewithin a second range of parameter values, and pixels shown in greenrepresent an acoustic parameter value within a third range of parametervalues, however, other display techniques are possible, includingadditional or alternative colors, patters, or the like. In variousembodiments, ranges of values can correspond to absolute ranges, such asintensity values between 10 dB and 20 dB, or can be relative ranges,such as intensity values between 90% and 100% of the maximum intensity.

As described elsewhere herein, in some embodiments, a display imageincluding electromagnetic image data and acoustic image data can includeboth a visual indication of an acoustic signal and an alphanumericrepresentation of one or more parameters associated with the acousticsignal. FIG. 16 shows an exemplary visualization showing bothnon-numeric information (e.g., palettization via parameter value ranges)and alphanumeric information. In the illustrated example, analphanumeric sound intensity value label is associated with each ofthree locations having palettized acoustic image data (e.g., intensitydata). As shown, acoustic signals have corresponding visual indicatorsrepresenting an acoustic parameter associated therewith (1602, 1604,1606), as well as alphanumeric information (1612, 1614, 1616,respectively). In an exemplary embodiment, the alphanumeric informationcan provide a numerical value, such as maximum intensity value,associated with the location at which the palettized acoustic data isdisplayed. In some examples, a user may select one or more locations atwhich to display the palettization and/or the alphanumeric data. Forexample, a user may choose to annotate a display image usingalphanumeric representations of an acoustic parameters associated withone or more acoustic signals within the scene.

In some examples, alphanumeric information can represent a plurality ofparameters (e.g., acoustic parameters) associated with an acousticsignal at a given location in a scene. FIG. 17 shows an examplevisualization including both non-numeric information (e.g.,palettization via parameter value ranges) and alphanumeric information.In the illustrated example, a sound intensity value and a correspondingfrequency value (e.g., an average frequency or a peak frequency) areshown in alphanumeric information 1712, 1714, 1716 associated with eachof three locations having palettized acoustic data (e.g., intensitydata) shown via indicators 1702, 1704, 1706, respectively. Similar to asdiscussed with respect to FIG. 16, the inclusion of various such data atvarious locations can be initiated by a user. For example, a user maychoose to annotate a display image using alphanumeric representations ofone or more acoustic parameters associated with one or more acousticsignals within the scene.

FIG. 18 shows another exemplary visualization showing both non-numericinformation (e.g., palettization via parameter value ranges) andalphanumeric information. In the illustrated example, a distancemeasurement is included with alphanumeric information 1812, 1814, 1816associated with each of three locations having palettized acoustic data(e.g., intensity data), shown via indicators 1802, 1804, 1806,respectively. Similar as discussed with respect to FIG. 16, theinclusion of various such data at various locations can be selected by auser, for example, as part of a display image annotation.

In some examples, non-numeric representations can be used to communicateinformation related to a plurality of acoustic parameters. For instance,FIG. 19 shows an exemplary visualization showing indicators 1902, 1904,1906 (in this case, circles) of different size and color representativeof different acoustic parameter values. In an exemplary embodiment, thesize of the indicator corresponds to the intensity of the acousticsignal at a given location, while the color of the indicator correspondsto a peak or average frequency. In an exemplary embodiment, indicatorsize can show relative values such that comparing the size of oneindicator to the size of another represents the relative differencebetween the represented acoustic parameter values at the locationsassociated with the indicators. Additionally or alternatively,alphanumeric information can be included to provide absolute or relativeacoustic parameter values.

In some embodiments, a colorized indicator can be used to represent theseverity of one or more detected acoustic signals and/or associatedacoustic parameters, such as an amount of deviation from a baselineparameter. FIG. 20 shows an exemplary visualization showing a pluralityof indicators 2002, 2004, 2006 having different colors indicative of theseverity indicated by acoustic signals from the corresponding locations.For instance, in an exemplary embodiment, a red indicator indicates acritical severity based on one more acoustic parameters (e.g., whencompared to a baseline, such as a baseline of typical operatingconditions), a yellow indicator indicates moderate severity, and a greenindicator represents minor severity. In other examples, other colorschemes or appearance characteristics (e.g., indicator transparency,indicator size, etc.) can be used to visually distinguish severity of anacoustic signal. In the illustrated example, indicator 2004 representsgreatest level of severity, indicator 2006 represents the next greatest,and indicator 2002 represents the least severe acoustic signal.

As described elsewhere herein, in various embodiments, one or moreacoustic parameters can be displayed on a visual representation of anacoustic scene, for example, by way of a palettized color or grayscaledisplay. In some embodiments, a system can be configured to identify oneor more locations in a scene meeting one or more acoustic conditions,such as an identified frequency range, intensity range, distance range,or the like. In some examples, various locations corresponding to anacoustic profile (e.g., satisfying a particular set of conditions orparameters) can be identified. Such identified locations can bepresented in a distinguishing manner from the acoustic image datapalettization scheme otherwise used in creating a display image. Forexample, FIG. 21 shows a scene including indicators at a plurality oflocations within the scene. Indicators 2101, 2102, 2103, 2104, and 2105are positioned within the scene. Indicators 2103, 2104, and 2105 includepalettized acoustic image data, for example, representing values of oneor more acoustic parameters corresponding to scale 2110. Indicators 2101and 2102 are shown having a unique presentation scheme distinguishablefrom the palettization scheme appearing at locations 2103, 2104, and2105. In such an embodiment, a user may quickly and easily identifythose locations in an image satisfying one or more desired conditions.In some such examples, a user may select the one or more desiredconditions for displaying in a distinguishing way based on a selectionof a range of values, for example, from a scale such as 2110.

Additionally or alternatively, a location meeting the conditions of aparticular sound profile can be presented with an icon representative ofthe met condition, such as a corresponding acoustic profile. Forexample, FIG. 22 shows a plurality of icons 2202, 2204, 2206, 2208positioned within a display image indication recognized acousticprofiles within the scene. Exemplary profiles as shown in FIG. 22include bearing wear, air leak, and electrical arcing. Such profiles canbe identified by acoustic signals satisfying a set of one or moreparameters associated with such profiles in order to be categorized intosuch profiles.

FIG. 23 shows another exemplary display showing acoustic data via aplurality of indicators 2302, 2304, and 2306 using concentric circlesand alphanumeric information representing acoustic intensity associatedwith each of the acoustic signals. As described elsewhere herein, insome examples, the size of an indicator can represent one or moreacoustic parameters present at the corresponding location. In someembodiments, indicators can be monochromatic, and indicate acousticparameters in one or more other ways, such as by indicator size, lineweight, line type (e.g., solid, dashed, etc.).

In some examples, a display can include alphanumeric information basedon a selection made by a user. For instance, in some embodiments, asystem (e.g., via a processor) can include information representing oneor more acoustic parameters of an acoustic signal located at aparticular location in response to a user selection of an indicator on adisplay (e.g., via a user interface) at such a location. FIG. 24 showsan example display image having an indicator and additional alphanumericinformation associated with the represented acoustic signal. In anexample, indicator 2402 on a display can be selected (e.g., representedvia crosshairs, which can be indicative of a selection, such as via atouch screen input) for further analysis. The display shows alphanumericinformation 2404 including a list of data associated with the locationcorresponding to the indicator, including a peak intensity andcorresponding frequency, a frequency range, a distance to locationmeasurement, and a level of criticality indicated by the acousticsignals from that location.

In some examples, a display image can include a plurality of indicatorsrepresenting a corresponding plurality of acoustic signals in the scene.In some embodiments, in such a case, a user may select one or moreindicators (e.g., via a touchscreen or other user interface), and inresponse to detecting the selection, the processor can presentadditional information regarding the acoustic signal. Such additionalinformation can include an alphanumeric of one or more acousticparameters. In some examples, such additional information can bedisplayed for multiple acoustic signals simultaneously. In otherexamples, such additional information for a given acoustic signal ishidden when another acoustic signal is selected.

As described elsewhere herein, in some examples, systems can include alaser pointer. In some examples, laser pointers can have a fixedorientation, or can have an adjustable pointing, for example,controllable via the processor. In some examples, the system can beconfigured to aim the laser pointer at a location in the target sceneassociated with a selected location in an image. FIG. 25A shows a system(in some examples, embodied as a handheld tool) including a display,such as the display shown in FIG. 24, in which an indicator 2502 isselected. A laser pointer 2504 emits a laser beam 2506 toward the scene,wherein the laser creates a laser spot 2508 in the scene correspondingto the location of the selected indicator 2502 in the image. This canhelp a user visualize the location of the selected and/or analyzedacoustic signals in the environment. In some embodiments, the laser spot2508 is detectable by an electromagnetic imaging tool, and is visible onthe display along with the displayed indicator 2502 and alphanumericinformation 2512 including acoustic parameter information. In someexamples, the acoustic imaging system is configured to detect or predictthe location of the laser in the scene, and provide a visual indication2510 of the laser location. FIG. 25B shows a display image such as thatshown in the system view of FIG. 25A.

In embodiments in which the laser pointer has a fixed orientation, theuser can view the display image having a visual indication of the laserlocation as feedback so that the user can adjust the pointing of thelaser to coincide with the selected acoustic signal.

As described elsewhere herein, in some embodiments, acoustic image datacan be combined with electromagnetic image data for presentation in adisplay image. In some examples, the acoustic image data can include anadjustable transparency such that various aspects of the electromagneticimage data are not completely obscured by FIG. 26 shows acoustic imagedata represented by an indicator 2602 at a location in the scene,wherein the indicator 2602 includes a gradient palettization scheme. Asystem can include a display device, which can be integral with orseparate from an acoustic imaging tool, configured to present displaydata including electromagnetic image data and acoustic image data.

In some embodiments, the device (e.g., a handheld acoustic imaging tool)can include a physical blending control 2614 (e.g., one or more buttons,knobs, sliders, etc., which can be included as part of a user interface)and/or a virtual blending control 2604, such as via a touchscreen orother virtually-implemented interface. In some embodiments, suchfunctionality can be provided by an external display device, such as asmartphone, tablet, computer, or the like.

FIG. 27 shows a virtual and/or physical blending control tools for adisplay image including a partially-transparent concentric circlepalettization scheme. Similar to as described with respect to FIG. 26,an indicator 2702 can represent an acoustic signal within a scene. Anacoustic imaging system can include a physical blending control 2714and/or a virtual blending control 2704 that can be used to adjust thetransparency of the acoustic image data (e.g., indicator 2702) withinthe display image.

Additionally or alternatively, physical and/or virtual interfaces can beused to adjust one or more display parameters. For instance, in someembodiments, one or more filters can be applied to selectively displayacoustic image data satisfying one or more conditions, such as describedelsewhere herein. FIG. 28 shows a scene including an indicator 2802having a gradient palettization indicating a location in the scenesatisfying one or more filters (e.g., having one or more acoustic orother parameters meeting one or more corresponding thresholds orpredetermined conditions). In various examples, the filtering can beselected and/or adjusted via a physical control 2814 (e.g., via one ormore buttons, knobs, switches, etc.) and/or a virtual control 2804(e.g., a touchscreen). In the illustrated example, the filter includesdisplaying acoustic image data for only those acoustic signals having anacoustic parameter (e.g., frequency) falling within a predefined range2806 of the acoustic parameter. As shown, the predefined range 2806 is asubset of possible filter ranges 2816. In some examples, a user mayadjust the limits of the predefined range 2806, for example, via virtual2804 or physical 2814 control to adjust the effects of the filter.

FIG. 29 shows a virtual and/or physical filter adjustment for a displayimage including a partially-transparent concentric circle palettizationscheme. As shown, indicator 2902 is shown within the scene based onacoustic parameters falling within a predetermined range 2906 of valuebased on a filter. The filter can be adjustable within a range of values2916, for example, via virtual 2904 and/or physical 2914 controls.

In some embodiments, a plurality of filters can be utilized forcustomizing a display image including palettized acoustic image data.FIG. 30 shows a display image showing a first indicator and a secondindicator. As described elsewhere herein, one or more filters can beapplied to the display image (e.g., via physical filter controls and/orvirtual filter controls) to customize the displayed data. In theillustrated example of FIG. 30, the filtering includes establishing afirst filter range 3006 and a second filter range 3008. In some example,filter ranges can be adjustable within a range of values 3016, forexample, via virtual 3004 and/or physical 3014 controls.

Such filter ranges can represent any of a variety of parameters, such asfrequency, amplitude, proximity, etc. As shown, the first filter rangeand the second filter range are each associated with a color (which, insome examples, can be adjustable by a user), and indicators 3002, 3012are positioned at locations in the image at which corresponding acousticsignals meet the one or more filter conditions associated with eachfilter range. As shown, the first indicator 3002 represents acousticsignals that satisfies the first filter range 3006 (shown in a darkershade), while the second indicator 3012 represents acoustic signals thatsatisfy the second filter range 3008 (shown in a lighter shade). Thus, auser may be able to quickly identify locations in a scene havingacoustic data satisfying a variety of conditions at once, while alsoidentifying which locations satisfy which conditions.

In some examples, a display device, such as an acoustic imaging tool oran external display device, can include a virtual keyboard as an inputdevice, such as shown in FIG. 31. FIG. 31 shows a display interfaceincluding an indicator 3102 representing one or more acoustic parametersof an acoustic signal in the scene. A virtual keyboard 3110 is includedin the display, which can be used to add alphanumeric information 3112to the display image. Utilizing such a virtual keyboard can allow a userto enter customize annotations, such as various inspection notes,labels, date/time stamps, or other data that can be stored with theimage. In various examples, the virtual keyboard can be used to add textthat is included in the image data and/or is appended to the image data,such as by being stored in metadata associated with a display image.

Various devices can be used to present a display image that includesvarious combinations of acoustic image data and other data, such asalphanumeric data, image data from one or more electromagneticspectrums, symbols, or the like. In some examples, a handheld acousticimaging tool can include a built-in display for presenting a displayimage. In other examples, information to be displayed, or data that isprocessed for generating a display (e.g., raw sensor data) can becommunicated to an external device for display. Such external devicescan include, for example, a smartphone, tablet, computer, wearabledevice, or the like. In some embodiments, the display image is presentedin combination with real-time electromagnetic image data (e.g., visiblelight image data) in an augmented realty-type display.

FIG. 32 shows a display embedded into eyewear 3210 that can be worn by auser. In some examples, eyewear can include one or more embedded imagingtools, such as described in U.S. Patent Publication No. 20160076937,entitled “DISPLAY OF IMAGES FROM AN IMAGING TOOL EMBEDDED OR ATTACHED TOA TEST AND MEASUREMENT TOOL,” and assigned to the assignee of theinstant application, relevant portions of which are incorporated hereinby reference. In some such examples, an integrated display can show areal-time display image 3220. For example, the display can showelectromagnetic image data (e.g., visible light image data)representative of the scene toward which the user is facing, and cansimultaneously display (e.g., via blending, overlay, etc.) one or moreadditional data streams, such as acoustic image data (e.g., includingindicator 3202) or the like, to provide added information to the user.In some embodiments, eyewear such as shown in FIG. 32 includes atransparent display screen, so that when no display image is provided tothe display, a user can view a scene directly with his or her eyesthrough eyewear rather than being presented with real-time visible lightimage data. In some such examples, additional data, such as acousticimage data, alphanumeric data, etc., can be displayed on theotherwise-transparent display in the user's field of view so that theuser views such data in addition to his or her view of the scene throughthe display.

As described elsewhere herein, in various examples, various datapresented in a display image can be combined in a variety of ways,including blending with other data streams (e.g., blending acousticimage data with visible light image data). In some examples, thestrength of the blending can vary between different locations within asingle display image. In some embodiments, a user can adjust theblending ratios of each of a plurality of locations (e.g., each of aplurality of indicators of detected acoustic signals) manually.Additionally or alternatively, blending can be a function of one or moreparameters, such as frequency, amplitude, proximity, etc.

In some embodiments, an acoustic imaging tool can be configured toidentify a degree to which the sensor array is pointing at each of aplurality of locations emitting detected acoustic signals and blendcorresponding acoustic image data with, for example, visible light imagedata, accordingly. FIG. 33A shows an exemplary display including a firstindicator 3302 and a second indicator 3304 representing acoustic signalsin an acoustic scene. In FIG. 33A, the acoustic sensor is pointing moredirectly at pipe 1, corresponding to the location of the first indicator3302, when compared to pipe 2, corresponding to the location of thesecond indicator 3304. As such, in the display scheme of FIG. 33A, thefirst indicator 3302 is displayed more prominently (e.g., has a higherblending coefficient or a lower transparency) than the second indicator3304. Conversely, in FIG. 33B the acoustic sensor is pointing moredirectly at pipe 2, corresponding to the location of the secondindicator 3304, when compared to pipe 1, corresponding to the locationof the first indicator 3302. As such, in the display scheme of FIG. 33B,the second indicator 3304 is displayed more prominently (e.g., has ahigher blending coefficient or a lower transparency) than the firstindicator 3302. In general, in some embodiments, an acoustic imagingsystem can determine a metric indicative of the degree to which thesensor is pointed at a given location (e.g., corresponding to anindicator in the acoustic image data) and adjust the blending ratioscorresponding to such locations accordingly (e.g., the greater thedegree of pointing corresponds to a higher blending ratio).

In some examples, the processor can save sound profiles detected in ascene. For instance, in an exemplary embodiment, a user can savedetected acoustic data (e.g., displayed as acoustic image data) as asound profile corresponding to one or more parameters. In some suchexamples, such a sound profile can be labeled according to and/orassociated with one or more characteristics of the scene, such as thepresence of an air leak, etc. Additionally or alternatively, predefinedsound profiles can be loaded into a system memory during factoryassembly of an acoustic imaging system and/or can be downloaded to orotherwise communicated to an acoustic imaging system.

In some examples, sound profiles can comprise one or more sounds presentin a scene. Various sound profiles can be defined by one or moreacoustic parameters, such as frequency, decibel level, periodicity,distance, or the like (e.g., an exemplary sound profile can include afrequency value within a predetermined range and a periodicity within apredetermined range associated with that profile). Sound profiles can bedefined by one or more sounds in a scene, and in some examples, each ofa plurality of sounds can be defined by one or more parameters, such asfrequency, decibel level, periodicity, distance, or the like. Multiplesounds in a given profile can be identified by like parameters (e.g.,two sounds each having a respective frequency range and periodicityrange) or can be identified by different parameters (e.g., one soundhaving a corresponding frequency range and another sound having acorresponding decibel level range and maximum distance value).

In some examples, the system can be configured to provide notificationsregarding a sound profile, such as if one or more sounds in an acousticscene correspond to a known sound profile. For instance, a notificationcan alert a user or technician of a recognized sound profile.Additionally or alternatively, the system can be configured to annotateacoustic image data, electromagnetic image data, and/or a display imagebased on a recognized sound profile. Notifications can include anaudible sound, visualization on the display screen, an LED light, or thelike.

In some examples, a system processor can be configured to analyze thecriticality of an acoustic signature, for example, in view of one ormore sound profiles. Correspondence between acoustic data (e.g., withrespect to a sound profile) and criticality of system operation can belearned, for example, based on machine learning and/or user inputs.

In some examples, a processor can be configured to compare data in anacoustic scene to one or more known sound profiles patterns to analyzethe scene, for example, for criticality of a detected signature. Invarious examples, the processor can be configured to notify a user basedon potentially criticality. This can be based on, for example, comparingan identified acoustic signature to one or more stored baselines,relative values compared to user-defined thresholds, and/or valuesdetermined automatically, such as via machine learning algorithms and/orartificial intelligence programming (e.g., based on historicalperformance and errors and corresponding historical acousticsignatures).

In various such examples, the processor can be configured to analyze anacoustic scene relative to one or more sound profiles and estimate theimpact of the discovered sound profile on the scene or object in thescene. Estimating the impact can include the potential criticality ofthe sound profile and/or the potential cost or loss of profit associatedwith the acoustic signature. In some examples, a system (e.g., via aprocessor) can be configured to recognize one or more air leaks in ascene (e.g., by comparing an acoustic signature to one or more soundprofiles associated with air leaks in an acoustic scene). In some suchexamples, a system can automatically calculate and/or report variousdata, such as a detected number of air leaks, severity of one or moresuch air leaks, and/or an estimated cost savings associated withrepairing such one or more leaks. In some examples, cost estimates canbe based on pre-programmed values and/or user input values associatedwith detected leaks. In an exemplary application, a system can beconfigured to determine a cost impact of a compressed air leak per unittime (e.g., per hour) if not properly remedied.

In an example, an acoustic imaging system can be configured to determinevarious information regarding an air leak in an acoustic scene, such asa pressure, orifice diameter, or leakage rate. In some examples, one ormore such values can be entered by a user, and the remaining value(s)can be calculated. For example, in an example, a user may enter apressure value associated with a particular air line, and using theinput pressure information, the system can be configured to determine anorifice diameter and leak rate based on acoustic data from the scene.Such determinations can be performed using, for example, a lookup tableand/or equation stored in memory.

In an example scenario, a detected sound profile can be associated withan air leak at 100 PSIG through a ¼″ diameter orifice. In someembodiments, an acoustic imaging system with access to such a storedsound profile can be programmed to recognize such a profile in anacoustic scene and estimate a cost per time associated with such a leak,for example, based on a lookup table. In a similar example, a detectedsound profile can be associated with a ¼″ diameter orifice based on anentered (e.g., via manual input) pressure of 100 PSIG. An acousticimaging system with access to such a stored sound profile can beprogrammed to recognize such a profile in an acoustic scene and estimatea cost per time associated with such a leak, for example, based on alookup table.

In some examples, an acoustic imaging system can be configured toperform a cost-savings analysis for fixing a plurality of detected leaksbased on an equation and/or a lookup table. In some examples, such anequation and/or lookup table can be stored in memory of or otherwiseaccessible by an acoustic imaging system for performing a cost analysisof an identified leak.

In an example implementation, an acoustic imaging system can beconfigured to characterize one or more leaks detected in an environment,for example, during an inspection of a scene or facility. Suchcharacterizations can be performed, for example, based on storedacoustic profiles corresponding to such leaks. Detected leaks (e.g.,with determined leakage rates) can be used to calculate a cost savingsof fixing such leaks. For example, an acoustic imaging system can beused to determine a number of leaks present and the leakage rateassociated with such leaks, and calculate a cost savings associated withsuch leaks.

In an example, cost savings can be calculated by multiplying a number ofleaks, leakage rate (cfm), an amount of energy associated with a leak(e.g., kW/cfm), number of operating hours, and cost per energy (e.g.,$/kWh), as shown in Equation (1) below:

Cost savings ($)=# leaks×leakage rate (cfm)×kW/cfm×# hours×$/kWh  (1)

An acoustic imaging system can be used to determine a number of leaksand the leakage rate (e.g., in cfm) associated with such leaks. Otherparameters may be preprogrammed into the system (e.g., energy per airgeneration in kW/cfm), accessed via a database (e.g., current cost ofenergy in $/kWh), or assumed by the system (e.g., an average number ofoperating hours). A system can be programmed to calculate a cost savingsassociate with fixing such leaks.

In an example, a system has 100 leaks of 1/32″ at 90 PSIG, 50 leaks of1/16″ at 90 PSIG, and 10 leaks of ¼″ at 100 PSIG. If assuming 7000annual operating hours, an aggregate electric rate of $0.05/kWh, andcompressed air generation requirement of approximately 18 kW/100 cfm,the cost savings associated with each leak, per Equation (1) is asfollows:

Cost savings from 1/32″ leaks=100×1.5×0.61×0.18×7000×0.05=$5,765

Cost savings from 1/16″ leaks=50×5.9×0.61×0.18×7000×0.05=$11,337

Cost savings from ¼″ leaks=10×104×0.61×0.18×7000×0.05=$39,967

As noted in the example, the savings from the elimination of just 10leaks of ¼″ account for almost 70% of the overall savings. As leaks areidentified, in some example, an acoustic imaging system can beconfigured to analyze the leaks and identify which leaks yield a highercost savings if fixed. In some examples, the system can rank orprioritize leaks with a higher cost savings and provide such rank orprioritization to a user. Additionally or alternatively, the system canbe configured to provide notifications to a user regarding the costsavings of one or more of the identified leaks.

FIG. 34 shows an exemplary display image that provides an indication toa user or technician regarding the potential criticality of and thepotential lost cost due to air leaks identified in the scene. In anexemplary embodiment, an acoustic imaging system (e.g., via a processor)detects the presence of leaks in the acoustic scene, for example, viarecognition of acoustic signals resembling known sound profilesassociated with leaks present in the acoustic scene. The system can beconfigured to analyze the detected acoustic signals to ascertain thecriticality of the identified leaks, for example, by determining one ormore sound profiles corresponding to the detected acoustic signals.

As described elsewhere herein, in some examples, acoustic image data canbe palettized according to a determined criticality. In someembodiments, criticality of acoustic data can be determined according toone or more sound profiles. For instance, a user may choose a leakdetection mode of operation, wherein the acoustic imaging systemanalyzes acoustic data with respect to one or more sound profilesassociated with leaks to determine criticality of acoustic data within ascene.

As described above, a system can be configured to calculate numericalcosts associated with one or more leaks, such as by way of one or moreequations and/or lookup tables. In an exemplary embodiment, a system canidentify, for instance, the size of and pressure associate with a leakbased on a sound profile associated with such a leak, and subsequentlycalculate a cost per unit time associated with such a leak by way of alookup table and/or equation.

In the illustrated example of FIG. 34, criticality and costs per unittime ($/year) are associated with each of a plurality of locations in anacoustic scene. In the illustrated embodiment, such locations are shownvia acoustic image data including a plurality of indicators 3410, 3412,3414 palettized according to criticality and combined with visible lightimage data to form a display image that provides leak criticalityinformation to a user for each location. Such criticality can bedetermined, for example, by analysis of an acoustic scene with respectto one or more corresponding sound profiles as described herein.

In various embodiments, leaks of varying degrees of criticality and/orpotential lost cost can be represented with acoustic image dataincluding of various color, shape, size, opacity, or the like.Similarly, one or more icons can be used to represent a specific leak(e.g., corresponding to a specific sound profile) or a range of cost orcriticality for the leaks. Additionally or alternatively, alphanumericinformation, such as a cost/year or the like, can be included proximatethe location of the corresponding leak. The display image of FIG. 34further includes a notification 3420 showing an approximate costassociated with each level of criticality. Such values can be calculatedbased on acoustic parameters associated with each acoustic signaldetected in the scene, for example, in view of a profile stored inmemory representing a particular air leak.

In some instances, additional contextual information is useful ornecessary in order to perform a proper analysis and reporting activitiesof the scene. For example, during some operations, when acousticallyimaging a scene, a user or technician may desire to record contextualinformation about the scene that is being inspected. In previoussystems, to perform this task, the user or technician must takephotographs with a separate camera or device, take written notes, orrecord notes using a separate device. Such notes must be manuallysynchronized with the data from the acoustic imaging device, potentiallyleading to errors in collection, recollection, and mismatching of data,and potentially leading to errors when performing an analysis orreporting.

In some embodiments, acoustic imaging systems according to the presentdisclosure can capture acoustic data of a target scene and thenassociate it with information relating to the target scene. Suchinformation related to the target scene can include details regardingone or more objects in the scene, the surroundings of the scene, and/orsurroundings of the location of the scene. In some embodiments, therelated information can be captured in the form of images, audiorecordings, or video recordings, and associated with acoustic datarepresentative of the scene (e.g., with the acoustic data itself, with adisplay image including corresponding acoustic image data, etc.). Insome examples, the related information is associated with the acousticdata to provide a greater understanding of what the informationrepresents. For example, the related information can include detailsregarding the target scene or object in the target scene.

In some examples, the systems can include one or more devices to gatherinformation related to an object in a scene or the scene in general. Theone or more such devices can include a camera, a positioning device, aclock, a timer, and/or various sensors, such as temperature sensors,electromagnetic sensors, humidity sensors, or the like. In someexamples, an acoustic imaging system can include a camera (e.g.,embedded into a housing of an acoustic imaging device) that can beconfigured for image and/or video acquisition to gather annotationinformation. The such an embedded camera can create photo or videoannotations which can be appended to or otherwise saved with theacoustic image data or any other data acquired by the acoustic imagingdevice.

In some examples, such an embedded camera can be configured to generateelectromagnetic imaging data that can be combined with acoustic imagedata for display, as described elsewhere herein. In some embodiments,the acoustic imaging system can be configured to save a variety ofinformation, such as acoustic data, acoustic image data, electromagneticimage data, annotation data (e.g., sensor data, image/video annotationdata, etc.) together with a time stamp. In some examples, acousticimaging systems can be configured to display and/or record relevantinformation both on the device and in software at a later time. Relevantinformation can be displayed on the display and/or saved as metadata,for example, with a saved display image or acoustic image file.

According to some embodiments, through the use of an embedded camera,image acquisition, or video acquisition device in an acoustic imagingsystem, such a device can be utilized to create photo or videoannotations which could be appended to and/or saved with the primaryacoustic data, the combined electromagnetic and acoustic image data,and/or any audio recordings.

In some examples, a user or technician can annotate the acoustic imagedata or other data collected by the acoustic imaging system shown on adisplay, for example, via a user interface (e.g., a touchscreen, one ormore buttons, etc.). For instance, in some examples, a user ortechnician can use on-display annotations to annotate the data whilerecording data or during playback of data collected at a previous time.

In some examples, the acoustic imaging system can annotate a displayimage (e.g., including acoustic image data and/or electromagnetic imagedata) via on-screen interaction from the system use. In some suchexamples, the device can save all relevant annotated information withthe primary acoustic image data, and forego any need for synchronizationor matching of data post hoc. Such implementation can reduce oreliminate human memory-related errors, which could result in incorrectpairing of primary and secondary data.

Various on-display annotations that can be added by a user (e.g., via auser interface) can include, but are not limited to on-display drawings(e.g., freehand), on-display text and writing, on-display shapecreation, on-display movement of objects, on-display placement ofpre-configured markers, on-display placement of pre-configured text,instructions, or notes, on-display placement of pre-configured shapes,on-display placement of pre-configured drawings or illustrations,on-display placement of pre-configured or programmed icons, oron-display visualization of one or more acoustic parameters.

In some examples, an acoustic imaging device can include a display,which can show collected (e.g., live or previously-collected) acousticimage data and/or electromagnetic image data. A user can annotate suchan image via controls or a touch interface integrated into display.

FIG. 35 shows an example of a user annotating the display image withon-display annotations. In the example of FIG. 35, three acousticsignals are shown at corresponding locations on the display image viapalettized indicators representing acoustic data associated with suchacoustic signals. As shown, in some examples, a user can annotate thedisplay image by drawing freeform shapes 3540 in order to highlight oremphasize portions of the display image. In this example, a userannotates the display image by encircling the centermost sound by afreeform drawing 3540. The illustrated example also shows additionalannotation information in the form of text 3542 that can be used toidentify or describe one or more components in the display image, suchas the component associated with the circled sound. For instance, in theillustrated example, a label of the apparent source of the circled sound(“Main line 2B-28”) is added to the display image. Such text can behand-written (e.g., via a touchscreen interface) or typed (e.g., via avirtual keyboard or a physical keyboard in communication with thesystem).

FIG. 36 shows an example of an annotated display image includinginstructions and relevant location information. In this figure, twosounds are shown on the display image, along with text information and agraphical indication instructing a user of a task to perform. A user canannotate the image as shown in order to instruct or remind a future userto perform one or more tasks while performing an acoustic inspection ofsuch a location.

When generating an annotated image such as that shown in FIG. 36, a usercan annotate the display image by inserting a pre-configured marker(e.g., arrow 3640) and/or handwritten text 3642. For instance, in theillustrated example, a user annotates the display image by inserting anarrow 3640 pointed towards a valve switch in the scene and textualinstructions 3642 to “shut off main valve first.”

FIG. 37 shows an example of a user annotating the display image withon-display annotations. In this example, a single acoustic signal isshown via a corresponding indicator on the display image. Theillustrated example shows a user annotating the display image byinserting a pre-programed icon 3740 into the display image, such as anair leak icon. Such an icon can be selected by a user and/or recommendedby the system based on one or more recognized characteristics sound(e.g., compared to a historical sound profile) and placed by theidentified sound. In addition, a user can annotate the image to identifyor label a possible source of the observed acoustic signals, such aswith a freeform label, a predefined shape, etc.

FIG. 38 shows an example of a user annotating the display image with anon-display annotation. In this figure, a single acoustic signal is shownvia a corresponding indicator on the display image. As shown, a user canannotate the display image by inserting a shape 3840 into the image. Inthe illustrated example, a user adds a box 3840 around a component onthe left-hand side of the display image to show a possible source of thesound from the acoustic image data. Such a box 3840 can be selected as apre-defined shape (e.g., a positionable and adjustable-size rectangle).Other shapes can be used and can be sized and positioned by a user toannotate the display image as desired.

In some examples, different types of labels can be combined on adisplay. For example, FIG. 39 shows an icon label 3942 positioned on adisplay image proximate an indicator in the acoustic image datarepresenting an acoustic signal within the scene as well as a rectangle3940 surrounding a component in the scene. Various combinations ofannotations are possible. In some embodiments, annotations are added toa display image automatically, for example, when a particular soundprofile is recognized in the scene. In some examples, a system mayprompt a user to annotate a display image in view of a recognized soundprofile.

Additionally or alternatively, in some examples, a user may choose toannotate an image by including visual or textual informationrepresenting acoustic parameters associated with an acoustic scene. Forexample, a user may, via a user interface (e.g., a touchscreen orphysical controls), select a particular type of display for showing oneor more acoustic parameters associated with the scene. A user maysimilarly annotate an image to include information regarding a cost orcriticality indication associated with a portion of the scene, such as adetected leak.

Annotations can include display features included in a liverepresentation of a display image and/or included in a single captureddisplay image, for example, stored in memory.

As described elsewhere herein, in various embodiments, acoustic imagingdevices can employ any number of different methods to display, localize,describe, and analyze detected sounds. Visualization methods couldinclude various types of colored shapes, icons, with various levels oftransparency adjustment to accommodate the visible background with whichthey are displayed. By simplifying parameter controls for acousticvisualizations on the device or making such controls more intuitive,users can more easily achieve better visualization results in less time,and with less training. Various methods of visualization parametercontrols can be implemented according to application and user needs.Many of these methods can be tailored to use by specific individuallevels of education and training in sound visualization andlocalization, thus providing a more adaptable device in various types ofapplications and organizations. As described elsewhere herein, in someexamples, a user may choose to annotate a display image by including aparticular data visualization scheme in the display image.

FIG. 40 shows an interface including a display image 4002 and amulti-parameter data visualization 4040 including plurality of frequencyranges on the right-hand side of the display image. In some examples, auser can select (e.g., via touch screen and/or other interface) one ormore frequency ranges from the plurality of displayed frequency ranges,for example, for filtering acoustic image data and displaying acousticimage data having frequency content (e.g., frequency content above acertain magnitude) associated with the selected frequency range(s). Inthe illustrated example, two frequency ranges 4042 and 4044 areselected. Corresponding indicators 4010 and 4012 in the display imageindicate locations in the scene having acoustic signals satisfying thecorresponding frequency ranges in the plurality of frequency rangesdisplayed on the screen. In the illustrated example, frequency range4042 and corresponding indicator 4010 are shown in a light shade, whilefrequency range 4044 and corresponding indicator 4012 are shown in adarker shade. In general, in some embodiments, an acoustic signal in thescene meeting a frequency range can be represented via an indicatorhaving a corresponding visual representation as the frequency range.

In general, the frequency ranges can be displayed in a variety of ways,such as a right-hand-justified, left-hand-justified, bottom-justified,or top-justified axis, or a central axis. In various embodiments, thefrequency ranges could be broken down into any degree of resolution,including every 1 kHz (e.g., 1 kHz-2 kHz; 2 kHz-3 kHz, etc.) or thelike. Such frequency ranges need not all be the same size or span thesame range of frequencies. In some examples, the physical size (e.g.,width) of the displayed frequency range in the multi-parameterrepresentation 4040 corresponds to one or more parameters, such as arelative amount of frequency content, amplitude of such frequencies inthe acoustic scene, proximity of such frequencies, etc.

In various embodiments, frequency ranges could be selected on a virtualcontrol through touch screen interaction and/or on a physical controlmechanism, such as a directional pad, where a user could scroll up,down, left, right through button pushes until the desired range bar(s)are highlighted, and then selected. In some examples, multiple rangescould be selected or deselected by the user.

In some embodiments, frequency bars (e.g., 4042, 4044) on the displayimage 4002 associated with various frequency ranges rise and fall withthe decibel level of the range in a real-time display image. In variousexamples, the decibel level of a range can be determined any number ofways, such as a peak decibel level, an average decibel level, a minimumdecibel level, a time-based average decibel level, or the like. In someembodiments, the decibel level associated with each frequency range canbe tracked over time. Tracking over time can include saving frequencyinformation at each of a plurality of times, such as at a giveninterval. Additionally or alternatively, tracking frequency data overtime can include tracking the peak decibel level observed at eachfrequency range over time (e.g., over a specific duration, operatingsession, etc.). Peak levels can be calculated in a variety of ways.

The display of frequency information can include, in addition tocurrent/recent frequency information, peak frequency data. FIG. 41 showsan interface including a display image 4102 and a multi-parameterrepresentation 4140 of frequency information, including plurality offrequency ranges positioned along a lower edge of the display image. Asshown in FIG. 41, frequency information, displayed on the bottom of thedisplay image, includes amplitude information (e.g., 4150) (measured indecibels in the example of FIG. 41) for a plurality of frequency ranges.In some examples, maximum decibel levels are displayed in one or morefrequency ranges. In some cases, the frequency ranges showing themaximum decibel levels are selectable by a user. Peak markers 4152 canremain on the multi-parameter representation 4140 indicating the maximumdecibel level, and can be represented in any number or possible ways,such as contrasting colors, bar caps, arrows, symbols, or numericvalues. In some examples, the maximum decibel levels shown in the datavisualization include a maximum value detected over a period of time,such as within the past 5 seconds, within the past 30 minutes, since thestart of a selected measurement, or the like. In some examples, a usercan reset the maximum value display so that previous maximum values aredisregarded.

In various embodiments, the frequency bands included in the displayimage can be adjusted by the user, or can be automatically determined bythe device with programmed algorithms or machine learning. In variousexamples, frequency bands can be equally sized and distributed, can beof different sizes, can be histogram equalized, or determined by anynumber of combined methods.

FIG. 42 shows a display image including frequency information for aplurality of frequency bands and peak values for a plurality offrequency bands similar to shown in FIG. 41. In the example of FIG. 42,a multi-parameter representation 4240 including intensity and frequencyinformation is shown on the right-hand portion of the screen, and theamplitude information (e.g., 4250) is shown horizontally with a rightjustified axis. Similar to FIG. 41, the multi-parameter representation4240 includes peak markers 4252 indicating peak amplitudes for one ormore frequency ranges over a period of time.

In some examples, decibel levels increase to the right and to the leftof a central axis, for instance, in a mirror image. Similarly, decibellevel peaks also can appear in a mirror image.

Such a mirrored decibel information with respect to a central axis isshown in FIG. 43. The display image of FIG. 43 includes multi-parameterrepresentation 4340 showing intensity information for a plurality offrequencies. Such a mirror axis representation can allow the user tobetter identify small changes or low decibel level changes, which may beimportant in frequency range selection.

Similar to FIGS. 41 and 42, the multi-parameter representation 4340 ofFIG. 43 includes peak markers 4352 indicating peak amplitudes for one ormore frequency ranges 4350 over a period of time.

In some embodiments, one or more frequency ranges can be palettized toindicate additional information regarding such frequency ranges, such asdecibel levels in such frequency ranges. FIG. 44 shows a multi-parameterrepresentation 4440 including palettized set of frequency ranges,wherein the palettization represents a decibel range into which eachfrequency range falls. For instance, in an exemplary embodiment,frequency ranges (e.g., 4450) shown in white fall within 0-20 dB,frequency ranges shown in a light gray (e.g., 4450) fall within 21-40dB, and frequency ranges shown in a dark gray fall within 41-100 dB. Insome examples, the palettization corresponds to relative values ratherthan absolute values, for example, wherein frequency ranges shown inwhite are considered to have “low” intensity, frequency ranges shown inlight gray are considered to have “moderate” intensity, and frequencyranges shown in dark gray are considered to have “high” intensity. In anexemplary embodiment, the bottom third of frequency ranges, in terms ofintensity, are labeled white, the middle third of frequency ranges, interms of intensity, are labeled light gray, and the highest third offrequency ranges, in terms of intensity, are shown in dark gray. Inanother example, frequency ranges having intensity up to one third ofthe maximum intensity are shown in white, frequency ranges havingintensity between one third and two thirds the maximum intensity areshown in light gray, and frequency ranges having intensity between twothirds of the maximum intensity and the maximum intensity are shown indark gray. In some examples, palettization schemes shown in thedisplayed frequency information can also be used in one or moreindicators present in acoustic image data. In general, any of a varietyof color or other visualization schemes (e.g., via various patterns,transparency, etc.) can be used.

In some examples, frequency ranges can be palettized in terms ofseverity of detected acoustic data in each frequency range. Forinstance, in some examples, frequency ranges shown in dark gray areconsidered to be critically severe, frequency ranges shown in light grayare considered to be moderately severe, and frequency ranges shown inwhite are considered to be exhibit minor severity. Similar to discussedabove, in some examples, one or more indicators present in acousticimage data can include similar palettization severity indications at oneor more locations in an acoustic scene.

FIG. 45 shows an example display image including a multi-parameterrepresentation 4540 showing different frequency ranges and indicators4510, 4512, 4514 palettized according to severity. In various examples,colors corresponding to different severity levels can be setautomatically by device or manually by user.

As described elsewhere herein, in some examples, frequency intensitydata can be saved or tracked over time. In some embodiments, intensityvs. time information can be displayed for each of one or more frequencyranges in a multi-parameter representation. FIG. 46 shows intensity (indB) vs. time trends for each of a plurality of frequency ranges (e.g.,4650) in multi-parameter representation 4640. For example, in someexamples, in addition to intensity vs. time, peak intensity informationcan also be displayed in the multi-parameter representation 4640 for oneor more of the one or more frequency ranges, such as via peak markers(e.g., 4652). As shown in FIG. 46, the multi-parameter representation4640 includes a time axis and represents, for each of the plurality ofacoustic frequencies or acoustic frequency ranges (e.g., 4650),intensity information corresponding to the acoustic frequency oracoustic frequency range over time.

Various processes as described herein can be embodied as anon-transitory computer-readable medium comprising executableinstructions for causing one or more processors for carrying out suchprocesses. Systems can include one or more processors configured toperform such processes, for example, based on instructions stored inmemory integral to or external from the processor. In some instances,various components can be distributed throughout the system. Forinstance, a system can include a plurality of distribute processors,each configured execute at least a portion of the overall processexecuted by a system. Additionally, it will be appreciated that variousfeatures and functions as described herein can be combined into a singleacoustic imaging system, for example, embodied as a handheld acousticimaging tool or a distributed system having various separate and/orseparable components.

Various functionalities of components described herein can be combined.In some embodiments, features described in this application can becombined with features described in the PCT application entitled“SYSTEMS AND METHODS FOR PROJECTING AND DISPLAYING ACOUSTIC DATA,”having attorney docket number 56581.178.2 and filed on Jul. 24, 2019,which is assigned to the assignee of the instant application and whichis incorporated herein by reference. In some embodiments, featuresdescribed in this application can be combined with features described inthe PCT application entitled “SYSTEMS AND METHODS FOR TAGGING ANDLINKING ACOUSTIC IMAGES,” having attorney docket number 56581.179.2 andfiled on Jul. 24, 2019, which is assigned to the assignee of the instantapplication and which is incorporated herein by reference. In someembodiments, features described in this application can be combined withfeatures described in the PCT application entitled “SYSTEMS AND METHODSFOR DETACHABLE AND ATTACHABLE ACOUSTIC IMAGING SENSORS,” having attorneydocket number 56581.180.2 and filed on Jul. 24, 2019, which is assignedto the assignee of the instant application and which is incorporatedherein by reference. In some embodiments, features described in thisapplication can be combined with features described in the PCTapplication entitled “SYSTEMS AND METHODS FOR REPRESENTING ACOUSTICSIGNATURES FROM A TARGET SCENE,” having attorney docket number56581.182.2 and filed on Jul. 24, 2019, which is assigned to theassignee of the instant application and which is incorporated herein byreference.

Various embodiments have been described. Such examples are non-limiting,and do not define or limit the scope of the invention in any way.

1. An acoustic analysis system comprising: an acoustic sensor array comprising a plurality of acoustic sensor elements, each of the plurality of acoustic sensor elements being configured to receive acoustic signals from an acoustic scene and output acoustic data based on the received acoustic signals; an electromagnetic imaging tool configured to receive electromagnetic radiation from a target scene and output electromagnetic image data representative of the received electromagnetic radiation; a user interface; a display; and a processor in communication with the acoustic sensor array, the electromagnetic imaging tool, the user interface, and the display, the processor being configured to: receive electromagnetic image data from the electromagnetic imaging tool; receive acoustic data from the acoustic sensor array; generate acoustic image data of a scene based on the received acoustic data; generate a display image comprising combined acoustic image data and electromagnetic image data; present the display image on the display; receive an annotation input from the user interface; and update the display image on the display based on the received annotation input.
 2. The system of claim 1, wherein: the received annotation comprises at least one of: a freestyle annotation, one or more icons, an alphanumeric annotation, and a predefined shape.
 3. (canceled)
 4. (canceled)
 5. The system of claim 1, wherein the processor is configured to: detect the location of an acoustic signal present in the acoustic scene; and wherein updating the display image on the display based on the received annotation input comprises: including an indicator in the display image positioned at the location of the acoustic signal; and adjusting a transparency of the indicator within the display image in response to a received annotation input associated with the transparency of the indicator, wherein adjusting the transparency of the indicator within the display image comprises adjusting a blending ratio corresponding to blending the electromagnetic imaging data and the acoustic image data.
 6. (canceled)
 7. The system of claim 1, wherein the processor is configured to: detect the location of an acoustic signal present in the acoustic scene; and determine an acoustic parameter associated with the acoustic signal; wherein the received annotation input comprises receiving a selection of a range of values associated with the acoustic parameter; and updating the display image on the display based on the received annotation input comprises including an indicator in the display image positioned at the location of the acoustic signal only if the acoustic parameter associated with the acoustic signal falls within the selected range of values associated with the acoustic parameter.
 8. (canceled)
 9. The system of claim 1, wherein the processor is configured to: detect the location of an acoustic signal present in the acoustic scene; and determine a distance to target associated with the acoustic signal; wherein the display image includes an indicator positioned at the location of the acoustic signal; and updating the display image on the display based on the received annotation input comprises including, proximate the indicator positioned at the location of the acoustic signal, an annotation indicating the distance to target associated with the acoustic signal.
 10. The system of claim 9, wherein the processor is configured to: determine one or more acoustic parameters associated with the acoustic signal; and determine a criticality associated with the acoustic signal based on a comparison of the acoustic parameter to a predetermined threshold or plurality of predetermined thresholds; wherein updating the display image on the display based on the received annotation input further comprises including, proximate the indicator positioned at the location of the acoustic signal, an annotation indicating the one or more acoustic parameters and the criticality associated with the acoustic signal.
 11. The system of claim 9, further comprising a laser distance sensor comprising a laser pointer configured to emit a laser toward the scene and provide information to the processor representative of the distance to the target scene; and wherein the display image further includes a laser indicator showing the position of the laser pointer within the target scene.
 12. (canceled)
 13. (canceled)
 14. The system of claim 1, wherein the processor is further configured to: detect a plurality of acoustic signals within the acoustic scene; determine a first acoustic parameter associated with each of the plurality of detected acoustic signals; and wherein receiving the annotation input from the user interface comprises receiving instructions to update the display image to include information representative of the first acoustic parameter; and updating the display image based on the received annotation input comprises including an indication on the display image representing the determined first acoustic parameter associated with each of the detected acoustic signals.
 15. The system of claim 14, wherein the indication on the display representing the determined first acoustic parameter associated with each of the detected acoustic signals comprises an isoacoustic indicator positioned at the location of each acoustic signal and including a single color, the single color representing a particular value or range of values associated with the first acoustic parameter.
 16. The system of claim 15, wherein the single color of each isoacoustic indicator represents a criticality level of the first acoustic parameter.
 17. The system of claim 15, wherein the processor is configured to determine a second acoustic parameter associated with each of the plurality of detected acoustic signals, and wherein each isoacoustic indicator represents a particular value or range of the first acoustic parameter and the second acoustic parameter.
 18. (canceled)
 19. The system of claim 14, wherein the indication comprises: for each of the plurality of detected acoustic signals, an indicator positioned on the display image at a location in the acoustic scene corresponding to the detected acoustic signal, the indicator having a visual characteristic representing the first acoustic parameter of the acoustic signal; and a multi-parameter representation showing a relationship between the first acoustic parameter and a second acoustic parameter within the acoustic scene.
 20. The system of claim 19, wherein the multi-parameter representation includes a time axis showing evolution of the relationship between the first acoustic parameter and the second acoustic parameter over time.
 21. (canceled)
 22. The system of claim 1, further comprising a memory including one or more sound profiles, each sound profile being defined by one or more acoustic parameters corresponding to one or more sounds; and wherein the processor is configured to: analyze the received acoustic data; and if the received acoustic data matches one of the sound profiles stored in memory, annotate the display image with an annotation associated with the sound profile; wherein the received annotation input comprises an instruction to detect a sound profile present in the acoustic scene.
 23. The system of claim 22, wherein the annotation comprises an indication of a detected condition of the acoustic scene based on the matched sound profile.
 24. The system of claim 23, wherein: the detected condition comprises an air leak; the processor is further configured to determine a cost associated with the air leak; and the indication comprises an alphanumeric annotation, the alphanumeric annotation comprising a representation of the cost associated with the air leak.
 25. (canceled)
 26. (canceled)
 27. (canceled)
 28. An acoustic analysis system comprising: an acoustic sensor array comprising a plurality of acoustic sensor elements, each of the plurality of acoustic sensor elements being configured to receive acoustic signals from an acoustic scene and output acoustic data based on the received acoustic signals; a memory comprising a sound profile, the sound profile comprising a first acoustic parameter and a first predetermined condition associated with the first acoustic parameter; a display; and a processor in communication with the acoustic sensor array, the display and the memory, the processor being configured to: receive acoustic data from the acoustic sensor array; determine if the received acoustic data matches the sound profile stored in the memory; and if the received acoustic data does match the sound profile stored in the memory, indicate, via an indication on the display, that the received acoustic data matches the sound profile stored in the memory, the indication comprising information related to the matched sound profile.
 29. The system of claim 28, wherein the determining if the received acoustic data matches the sound profile comprises determining if the received acoustic data includes an acoustic signal satisfying the first predetermined condition associated the first acoustic parameter.
 30. The system of claim 28, wherein the sound profile comprises a second acoustic parameter and a second predetermined condition associated with the second acoustic parameter.
 31. The system of claim 30, wherein the received acoustic data matches the sound profile if the received acoustic data includes an acoustic signal satisfying the first predetermined condition associated with the first acoustic parameter and the second predetermined condition associated with the second acoustic parameter.
 32. The system of claim 31, wherein the first acoustic parameter comprises frequency, the first predetermined condition comprises a range of frequencies, the second acoustic parameter comprises intensity, and the second predetermined condition comprises a threshold intensity, such that: the received acoustic data matches the sound profile if the received acoustic data includes an acoustic signal having a frequency within the range of frequencies and an intensity greater than the threshold intensity.
 33. The system of claim 30, wherein the received acoustic data matches the sound profile if the received acoustic data includes a first acoustic signal satisfying the first predetermined condition associated with the first acoustic parameter and a second acoustic signal second predetermined condition associated with the second acoustic parameter.
 34. An acoustic analysis system comprising: an acoustic sensor array comprising a plurality of acoustic sensor elements, each of the plurality of acoustic sensor elements being configured to receive acoustic signals from an acoustic scene and output acoustic data based on the received acoustic signals; an electromagnetic imaging tool configured to receive electromagnetic radiation from a target scene and output electromagnetic image data representative of the received electromagnetic radiation, the electromagnetic imaging tool being configured to detect electromagnetic radiation; a display; and a processor in communication with the acoustic sensor array, the electromagnetic imaging tool, and the display, the processor being configured to: receive electromagnetic image data from the electromagnetic imaging tool; receive acoustic data from the acoustic sensor array; generate acoustic image data of a scene based on the received acoustic data; generate a display image comprising combined acoustic image data and electromagnetic image data; and present the display image on the display, wherein the display image includes: an indicator positioned on the display image at a location in the acoustic scene corresponding to a detected acoustic signal, the indicator having a visual characteristic representing an acoustic parameter of the acoustic signal; and a multi-parameter representation that includes intensity information representative of the received acoustic data corresponding to each of a plurality of acoustic frequencies or acoustic frequency ranges.
 35. The system of claim 34, wherein the multi-parameter representation includes a time axis and represents, for each of the plurality of acoustic frequencies or acoustic frequency ranges, the intensity information corresponding to the acoustic frequency or acoustic frequency range over time.
 36. The system of claim 34, further comprising a user interface in communication with the processor, and wherein the processor is configured to: receive, via the user interface, a selection of one or more of the plurality of acoustic frequencies or acoustic frequency ranges; and determine locations in the acoustic scene having acoustic signals including frequency content within the selected one or more acoustic frequencies or acoustic frequency ranges; and wherein the generated acoustic image data includes information only in the determined locations having acoustic signals including frequency content within the selected one or more acoustic frequencies or acoustic frequency ranges.
 37. The system of claim 34, wherein the multi-parameter representation includes, for each of the plurality acoustic frequencies or acoustic frequency ranges, an indication of a current acoustic intensity value and a representation of a maximum acoustic intensity value.
 38. (canceled)
 39. The system of claim 34, wherein the indicator and/or the multi-parameter representation includes criticality information representing a criticality of acoustic data in a location within the acoustic scene and/or at a predetermined acoustic frequency or range of acoustic frequencies within the acoustic scene. 